Scaling & Learning Theory
Scaling laws and generalization behaviour of deep networks.
A topic in the Science corpus, drawn from 334 papers.
Trusted voices in Scaling & Learning Theory
Ordered by the trust Scaling & Learning Theory's own network concentrates on each researcher, most first.
- Vladimir Naumovich Vapni — trusted by 1,806 researchers here
- Yuhai Wu — trusted by 994 researchers here
- J. Urgen Schmidhuber — trusted by 485 researchers here
- Peter L. Bartlett — trusted by 218 researchers here
- Yoshua Bengio — trusted by 235 researchers here
- Surya Ganguli — trusted by 189 researchers here
- Kenji Kawaguchi — trusted by 158 researchers here
- Martin H. Anthony — trusted by 150 researchers here
- Yuanzhi Li — trusted by 132 researchers here
- Andrew M. Saxe — trusted by 179 researchers here
- Jonathan Frankle — trusted by 178 researchers here
- Michael Carbin — trusted by 173 researchers here
Most-cited papers in Scaling & Learning Theory
Ordered by the recorded citation count from the source corpus; a citation is not an endorsement.
- The Nature of Statistical Learning Theory (2000) — 45,800 citations
- Statistical Learning Theory (2021) — 22,124 citations
- Deep learning in neural networks: An overview (2014) — 17,623 citations
- Power-Law Distributions in Empirical Data (2007) — 9,837 citations
- Scaling Laws for Neural Language Models (2020, arXiv.org) — 8,567 citations
- Machine learning: Trends, perspectives, and prospects (2015) — 8,448 citations
- PaLM: Scaling Language Modeling with Pathways (2022, Journal of machine learning research) — 8,214 citations
- Understanding deep learning requires rethinking generalization (2016) — 5,191 citations
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018) — 4,345 citations
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks (2018) — 4,156 citations
- Training Compute-Optimal Large Language Models (2022, Advances in Neural Information Processing Systems 35) — 3,535 citations
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (2016) — 3,531 citations
- Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (2019) — 3,265 citations
- Shortcut learning in deep neural networks (2020) — 3,155 citations
- Understanding deep learning (still) requires rethinking generalization (2021) — 2,784 citations
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017) — 2,747 citations
- On the Spectral Bias of Neural Networks (2018) — 2,472 citations
- Visualizing the Loss Landscape of Neural Nets (2017) — 2,377 citations
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013) — 2,139 citations
- Reconciling modern machine-learning practice and the classical bias–variance trade-off (2018) — 2,107 citations
- Deep learning and the information bottleneck principle (2015) — 2,084 citations
- Averaging Weights Leads to Wider Optima and Better Generalization (2018) — 2,084 citations
- The Tradeoffs of Large Scale Learning (2007) — 1,747 citations
- A Convergence Theory for Deep Learning via Over-Parameterization (2018) — 1,651 citations
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher (2021) — 1,640 citations
- Reproducible Scaling Laws for Contrastive Language-Image Learning (2022) — 1,494 citations
- Spectrally-normalized margin bounds for neural networks (2017) — 1,473 citations
- Scaling Vision Transformers (2021) — 1,462 citations
- Train faster, generalize better: Stability of stochastic gradient descent (2015) — 1,448 citations
- Exploring Generalization in Deep Learning (2017) — 1,437 citations
- On the Number of Linear Regions of Deep Neural Networks (2014) — 1,419 citations
- Minimum Description Length Principle. (2010) — 1,403 citations
- Scaling learning algorithms towards AI (2007) — 1,387 citations
- Wide neural networks of any depth evolve as linear models under gradient descent (2019) — 1,337 citations
- Deep double descent: where bigger models and more data hurt (2019) — 1,162 citations
- The Implicit Bias of Gradient Descent on Separable Data (2017) — 1,145 citations
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019) — 1,081 citations
- On Exact Computation with an Infinitely Wide Neural Net (2019) — 1,059 citations
- Deep Learning Scaling is Predictable, Empirically (2017) — 1,031 citations
- The Expressive Power of Neural Networks: A View from the Width (2017) — 1,013 citations
- Deep Learning without Poor Local Minima (2016) — 1,010 citations
- Large Batch Training of Convolutional Networks (2017) — 967 citations
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data (2017) — 967 citations
- Benign overfitting in linear regression (2019) — 961 citations
- The average distances in random graphs with given expected degrees (2002) — 929 citations
- Sharp Minima Can Generalize For Deep Nets (2017) — 918 citations
- Scaling Vision Transformers to 22 Billion Parameters (2023) — 911 citations
- On the Expressive Power of Deep Neural Networks (2016) — 897 citations
- The Minimum Description Length Principle (Adaptive Computation and Machine Learning) (2007) — 894 citations
- Surprises in High-Dimensional Ridgeless Least Squares Interpolation (2019) — 892 citations
Most active researchers
By papers in this topic: a volume measure, not trust.
- Nati Srebro — 9 papers
- Yoshua Bengio — 8 papers
- Surya Ganguli — 8 papers
- Behnam Neyshabur — 8 papers
- Jascha Narain Sohl-Dickstein — 7 papers
- Jeffrey Pennington — 7 papers
- Yi Tay — 7 papers
- Sanjeev Arora — 7 papers
- Yann LeCun — 7 papers
- Oriol Vinyals — 7 papers
- Levent Sagun — 7 papers
- Greg Yang — 7 papers