Kaifeng Lyu 吕凯风

I believe that a good theory should come from practice and be able to advance practice: it should start from important real-world phenomena or problems, explain or solve them theoretically, and then be applied to guide the practice. Based on this philosophy, I conduct research with both theory and experiments, aiming to develop solid foundations for modern machine learning and make AI in the era of large models more efficient, safe, and reliable.

Below is a list of topics I am actively thinking about/working on:

1. Training Dynamics of Neural Networks: The training dynamics of neural networks are extremely complex, but are there any universal laws that we can predict? What are the phase transitions that make model performance unpredictable? Can we even predict these phase transitions? Our previous works include:

   • Theoretical analyses for how to scale hyperparameters in distributed training [1], [2], [3]
   • What is the best learning rate schedule for large models? [4]
   • The amount of knowledge learned from mixed data does not grow linearly with model size, but may exhibit phase transitions [5]
   • Grokking: Why does the test accuracy of a neural network suddenly improve after training on the training set for a long time? [6]
   • How does the normalization layer help neural network training? [7], [8], [9]
   • Implicit Bias of GD: Even if you do not add regularization to your network, gradient descent will add it for you [10], [11], [12]

2. Generalization Paradigms of Foundation Models: Modern large foundation models integrate various unsupervised, supervised and reinforcement learning paradigms, and can achieve amazing generalization ability after training on large amounts of data. How do we understand these new generalization paradigms? How do we combine algorithms, architectures, and data better to enhance the model's capabilities? Can these understandings guide us to better select, mix, and even generate data? Our previous works include:

   • How does architectures affect Chain-of-Thought reasoning performance? [13]
   • Weak-to-strong Generalization: Does the phenomenon that GPT-4 supervised by GPT-2 outperforms GPT-2 also happen in simpler models? [14]

3. Foundations of AI Safety/Alignment: I am also interested in AI safety and alignment. Machine learning usually optimizes the performance of a model in the “average case”, but AI safety issues can be exposed in extreme cases. What is the fundamental reason for model failures in the extreme case? What are the limitations of current AI alignment methods, and what are the security issues that cannot be completely avoided? In the long run, can we find a systematic method like cryptography to solve a large class of AI safety problems once and for all? Our previous works include:

   • Current RLHF-based safety alignment is very shallow, shallow enough that the difference in output token distribution between base and aligned models is only a few tokens deep [15]
   • Fine-tuning a well-aligned model may degrade its safety, but simple data format adjustments can mitigate this problem [16]
   • Theoretically, why does a neural network not have adversarial robustness? [17], [18]