Backward Feature Correction: How Deep Learning Performs Deep Learning

By Zeyuan Allen-Zhu, Yuanzhi Li. 2020

125 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

1. Yuanzhi Li
2. Zeyuan Allen-Zhu
3. Hongyang Zhang
4. Tengyu Ma
5. Weizhu Chen
6. Andrea Montanari
7. Yelong Shen
8. J. Edward Hu

In Model Architectures

1. Yuanzhi Li
2. Zeyuan Allen-Zhu
3. Tengyu Ma
4. Andrea Montanari
5. Theodor Misiakiewicz
6. Song Mei
7. Behrooz Ghorbani
8. Hongyang Zhang

In Optimization & Training

1. Yuanzhi Li
2. Zeyuan Allen-Zhu
3. Tengyu Ma
4. Hongyang Zhang
5. Andrea Montanari
6. Song Mei
7. Theodor Misiakiewicz
8. J. Edward Hu

Citations in the corpus

• LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “In theory literature, it is known that neural networks outperform other classical learning methods, including the corresponding (finite-width) neural tangent kernels [5, 22] when the underlying concept class has certain low-rank structure [2, 3, 11].”
• When do neural networks outperform kernel methods? (2020, background): “target functions f∗ that can be represented and possibly efficiently learnt using neural networks, but not in the corresponding RKHS [32, 6, 17, 16, 1, 2].”
• Feature Purification: How Adversarial Training Performs Robust Deep Learning (2020, background): “Other related works prove lower bounds for kernel method in the regression case [99, 100].”
• What Can ResNet Learn Efficiently, Going Beyond Kernels? (2019, background): “In a follow-up work [2], this theory of hierarchical learning is significantly strengthened to further incorporate the “backward feature correction” when training deep neural networks.”
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, methodology): “We also conjecture that one can prove this concept class is not efficiently learnable by kernel methods in general, using the recent development of kernel lower bounds [2, 4].”
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK (2020, background): “Allen-Zhu and Li (2019a, 2020a) show that over-parametrized neural networks can learn certain concept class more efficient than any kernel method.”