Statistical Mechanics of Deep Learning

By Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam Schoenholz, Jascha Narain Sohl-Dickstein, Surya Ganguli. 2020

53 citations recorded by the source corpus.

The recent striking success of deep neural networks in machine learning raises profound questions about the theoretical principles underlying their success. For example, what can such deep networks compute? How can we train them? How does information propagate through them? Why can they generalize? And how can we teach them to imagine? We review recent work in which methods of physical analysis rooted in statistical mechanics have begun to provide conceptual insights into these questions. These insights yield connections between deep learning and diverse physical and mathematical topics, including random landscapes, spin glasses, jamming, dynamical phase transitions, chaos, Riemannian geometry, random matrix theory, free probability, and nonequilibrium statistical mechanics. Indeed, the fields of statistical mechanics and machine learning have long enjoyed a rich history of strongly coupled interactions, and recent advances at the intersection of statistical mechanics and deep learning suggest these interactions will only deepen going forward.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

Citations in the corpus

• Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, unknown): “In the past, statistical mechanics provided a useful theoretical framework for such typical-case analyses for various algorithms 23 – 32 .”
• If deep learning is the answer, what is the question? (2020, unknown): “…neuroscientists, physicists and computer scientists have paved the way for a new approach that uses idealized neural network models to understand the mathematical principles by which they learn 137 , and deploys the results to predict or explain phenomena in psychology or neuroscience 138 .”