Jamming in Multilayer Supervised Learning Models.

By Silvio Franz, Sungmin Hwang, Pierfrancesco Urbani. 2018

46 citations recorded by the source corpus.

Critical jamming transitions are characterized by an astonishing degree of universality. Analytic and numerical evidence points to the existence of a large universality class that encompasses finite and infinite dimensional spheres and continuous constraint satisfaction problems (CCSP) such as the nonconvex perceptron and related models. In this Letter we investigate multilayer neural networks (MLNN) learning random associations as models for CCSP that could potentially define different jamming universality classes. As opposed to simple perceptrons and infinite dimensional spheres, which are described by a single effective field in terms of which the constraints appear to be one dimensional, the description of MLNN involves multiple fields, and the constraints acquire a multidimensional character. We first study the models numerically and show that similarly to the perceptron, whenever jamming is isostatic, the sphere universality class is recovered, we then write the exact mean-field equations for the models and identify a dimensional reduction mechanism that leads to a scaling regime identical to the one of spheres.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Statistical and Nonlinear Physics, Statistical Learning Theory, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Statistical and Nonlinear Physics

In Statistical Learning Theory

In AI for Physical Sciences

Citations in the corpus

Scaling description of generalization with number of parameters in deep learning (2019, background): “In the perceptron, the simplest network without hidden layers, the cusp in the test error at the jamming point is also observed and predicted analytically [50, 51, 52, 53, 54, 55].”
The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “The manuscript [75], which appeared at the same time than ours, shows that the critical properties of the jamming transition found for the non-convex perceptron [34] hold more generally in some shallow networks.”
A jamming transition from under- to over-parametrization affects generalization in deep learning (2018, background): “The manuscript [40], which appeared at the same time as ours, shows that the critical properties of the jamming transition found for the non-convex perceptron [41] hold more generally in some shallow networks.”
Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm (2019, methodology): “The Teacher-Student paradigm has been broadly used to study supervised learning (Saad and Solla, 1995; Monasson and Zecchina, 1995; Opper and Saad, 2001; Engel and Van den Broeck, 2001; Zdeborová and Krzakala, 2016; Barbier et al., 2019; Gabrié et al., 2018; Aubin et al., 2018; Franz et al., 2018).”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “The manuscript [35], which appeared at the same time than ours, shows that the critical properties of the jamming transition found for the non-convex perceptron [14] hold more generally in some shallow networks.”