Statistical mechanics of complex neural systems and high dimensional data

By Madhu S. Advani, Subhaneil Lahiri, Surya Ganguli. 2013

79 citations recorded by the source corpus.

Recent experimental advances in neuroscience have opened new vistas into the immense complexity of neuronal networks. This proliferation of data challenges us on two parallel fronts. First, how can we form adequate theoretical frameworks for understanding how dynamical network processes cooperate across widely disparate spatiotemporal scales to solve important computational problems? Second, how can we extract meaningful models of neuronal systems from high dimensional datasets? To aid in these challenges, we give a pedagogical review of a collection of ideas and theoretical methods arising at the intersection of statistical physics, computer science and neurobiology. We introduce the interrelated replica and cavity methods, which originated in statistical physics as powerful ways to quantitatively analyze large highly heterogeneous systems of many interacting degrees of freedom. We also introduce the closely related notion of message passing in graphical models, which originated in computer science as a distributed algorithm capable of solving large inference and optimization problems involving many coupled variables. We then show how both the statistical physics and computer science perspectives can be applied in a wide diversity of contexts to problems arising in theoretical neuroscience and data analysis. Along the way we discuss spin glasses, learning theory, illusions of structure in noise, random matrices, dimensionality reduction and compressed sensing, all within the unified formalism of the replica method. Moreover, we review recent conceptual connections between message passing in graphical models, and neural computation and learning. Overall, these ideas illustrate how statistical physics and computer science might provide a lens through which we can uncover emergent computational functions buried deep within the dynamical complexities of neuronal networks.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Scaling & Learning Theory, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Scaling & Learning Theory

In Probabilistic & Bayesian Methods

Citations in the corpus

High-dimensional dynamics of generalization error in neural networks (2017, background): “error dynamics of various simple training scenarios in the \high-dimensional" regime where the number of samples P and parameters N are both large ( P;N ! 1 ), but their ratio = P=N is nite (e.g Advani, Lahiri, and Ganguli, 2013). We start with simple models, where analytical results are possib”
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks (2020, methodology): “Following (Advani et al., 2013), we represent the determinant as a Gaussian integral and replicate To perform the average, we will resort to a Gaussian approximation.”
Statistical Mechanics of Optimal Convex Inference in High Dimensions (2016, background): “See also [10,21] for general reviews on replica theory in highdimensional inference problems.”
Beyond neural scaling laws: beating power law scaling via data pruning (2022, background): “[22] Madhu Advani, Subhaneil Lahiri, and Surya Ganguli.”
A high-bias, low-variance introduction to Machine Learning for physicists (2018, unknown): “Much of this research has focused on understanding high-dimensional linear regression and compressed sensing (Donoho, 2006) (see (Advani et al., 2013; Zdeborová and Krzakala, 2016) for accessible reviews for physicists).”
Triple descent and the two kinds of overfitting: where and why do they appear? (2020, unknown): “The spectral gap (distance of the left edge of the spectrum to zero) closes at N=P , causing the nonlinear peak [46], but remains finite at N=D.”
On simplicity and complexity in the brave new world of large-scale neuroscience. (2015, background): “They reveal the beautiful fact that we can recover a lot of information about large objects (vectors or matrices) using surprisingly few measurements when we have seemingly weak prior information, like sparsity, or low-rank structure (see [53, 54] for reviews in a neuroscience context).”