Scaling & Learning Theory

Scaling laws and generalization behaviour of deep networks.

A topic in the Science corpus, drawn from 334 papers.

Trusted voices in Scaling & Learning Theory

Ordered by the trust Scaling & Learning Theory's own network concentrates on each researcher, most first.

  1. Vladimir Naumovich Vapni — trusted by 1,806 researchers here
  2. Yuhai Wu — trusted by 994 researchers here
  3. J. Urgen Schmidhuber — trusted by 485 researchers here
  4. Peter L. Bartlett — trusted by 218 researchers here
  5. Yoshua Bengio — trusted by 235 researchers here
  6. Surya Ganguli — trusted by 189 researchers here
  7. Kenji Kawaguchi — trusted by 158 researchers here
  8. Martin H. Anthony — trusted by 150 researchers here
  9. Yuanzhi Li — trusted by 132 researchers here
  10. Andrew M. Saxe — trusted by 179 researchers here
  11. Jonathan Frankle — trusted by 178 researchers here
  12. Michael Carbin — trusted by 173 researchers here

Most-cited papers in Scaling & Learning Theory

Ordered by the recorded citation count from the source corpus; a citation is not an endorsement.

  1. The Nature of Statistical Learning Theory (2000) — 45,800 citations
  2. Statistical Learning Theory (2021) — 22,124 citations
  3. Deep learning in neural networks: An overview (2014) — 17,623 citations
  4. Power-Law Distributions in Empirical Data (2007) — 9,837 citations
  5. Scaling Laws for Neural Language Models (2020, arXiv.org) — 8,567 citations
  6. Machine learning: Trends, perspectives, and prospects (2015) — 8,448 citations
  7. PaLM: Scaling Language Modeling with Pathways (2022, Journal of machine learning research) — 8,214 citations
  8. Understanding deep learning requires rethinking generalization (2016) — 5,191 citations
  9. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018) — 4,345 citations
  10. Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018) — 4,156 citations
  11. Training Compute-Optimal Large Language Models (2022, Advances in Neural Information Processing Systems 35) — 3,535 citations
  12. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016) — 3,531 citations
  13. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (2019) — 3,265 citations
  14. Shortcut learning in deep neural networks (2020) — 3,155 citations
  15. Understanding deep learning (still) requires rethinking generalization (2021) — 2,784 citations
  16. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017) — 2,747 citations
  17. On the Spectral Bias of Neural Networks (2018) — 2,472 citations
  18. Visualizing the Loss Landscape of Neural Nets (2017) — 2,377 citations
  19. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013) — 2,139 citations
  20. Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018) — 2,107 citations
  21. Deep learning and the information bottleneck principle (2015) — 2,084 citations
  22. Averaging Weights Leads to Wider Optima and Better Generalization (2018) — 2,084 citations
  23. The Tradeoffs of Large Scale Learning (2007) — 1,747 citations
  24. A Convergence Theory for Deep Learning via Over-Parameterization (2018) — 1,651 citations
  25. Scaling Language Models: Methods, Analysis & Insights from Training Gopher (2021) — 1,640 citations
  26. Reproducible Scaling Laws for Contrastive Language-Image Learning (2022) — 1,494 citations
  27. Spectrally-normalized margin bounds for neural networks (2017) — 1,473 citations
  28. Scaling Vision Transformers (2021) — 1,462 citations
  29. Train faster, generalize better: Stability of stochastic gradient descent (2015) — 1,448 citations
  30. Exploring Generalization in Deep Learning (2017) — 1,437 citations
  31. On the Number of Linear Regions of Deep Neural Networks (2014) — 1,419 citations
  32. Minimum Description Length Principle. (2010) — 1,403 citations
  33. Scaling learning algorithms towards AI (2007) — 1,387 citations
  34. Wide neural networks of any depth evolve as linear models under gradient descent (2019) — 1,337 citations
  35. Deep double descent: where bigger models and more data hurt (2019) — 1,162 citations
  36. The Implicit Bias of Gradient Descent on Separable Data (2017) — 1,145 citations
  37. Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019) — 1,081 citations
  38. On Exact Computation with an Infinitely Wide Neural Net (2019) — 1,059 citations
  39. Deep Learning Scaling is Predictable, Empirically (2017) — 1,031 citations
  40. The Expressive Power of Neural Networks: A View from the Width (2017) — 1,013 citations
  41. Deep Learning without Poor Local Minima (2016) — 1,010 citations
  42. Large Batch Training of Convolutional Networks (2017) — 967 citations
  43. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data (2017) — 967 citations
  44. Benign overfitting in linear regression (2019) — 961 citations
  45. The average distances in random graphs with given expected degrees (2002) — 929 citations
  46. Sharp Minima Can Generalize For Deep Nets (2017) — 918 citations
  47. Scaling Vision Transformers to 22 Billion Parameters (2023) — 911 citations
  48. On the Expressive Power of Deep Neural Networks (2016) — 897 citations
  49. The Minimum Description Length Principle (Adaptive Computation and Machine Learning) (2007) — 894 citations
  50. Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2019) — 892 citations

Most active researchers

By papers in this topic: a volume measure, not trust.