Weight space structure and internal representations: A direct approach to learning and generalization in multilayer neural networks.

By Rémi Monasson, Riccardo Zecchina. 1995

Subfields: Representation & Self-Supervised Learning, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Scaling & Learning Theory

Citations in the corpus

The jamming transition as a paradigm to understand the loss landscape of deep neural networks (2018, background): “On the other hand, one can handcraft neural networks that fit even structure-less, random data with a rather small number of parameters N ∼ P [14–17].”
Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “Local entropy, on the other hand, exploits wide minima which have been shown to exist in a variety of learning problems (Monasson & Zecchina, 1995; Cocco et al., 1996).”
Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm (2019, methodology): “The Teacher-Student paradigm has been broadly used to study supervised learning (Saad and Solla, 1995; Monasson and Zecchina, 1995; Opper and Saad, 2001; Engel and Van den Broeck, 2001; Zdeborová and Krzakala, 2016; Barbier et al., 2019; Gabrié et al., 2018; Aubin et al., 2018; Franz et al., 2018).”
Statistical mechanics of complex neural systems and high dimensional data (2013, methodology): “Statistical mechanics approaches have been used to analyze memory [144, 145, 146] and generalization [147, 148, 149] in more sophisticated multilayered networks.”
The committee machine: computational to statistical gaps in learning a two-layers neural network (2018, background): “1 Introduction While the traditional approach to learning and generalization follows the Vapnik-Chervonenkis [1] and Rademacher [2] worst-case type bounds, there has been a considerable body of theoretical work on calculating the generalization ability of neural networks for data arising from a prob”
Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning (2019, methodology): “[38] Rémi Monasson and Riccardo Zecchina.”
Statistical Mechanics of Learning : Generalization (2002, background): “A more complicated scheme called replica symmetry breaking (Mézard et al,1987) for continuing the matrices qab to noninteger dimensions can be necessary if the version space of the learning problem is e.g. disconnected or not convex (Monasson and Zecchina, 1995).”
Shaping the learning landscape in neural networks around wide flat minima (2019, background): “The geometrical structure of the absolute minima of the error loss function is known (20) to be composed of multiple regions, each in one to one correspondence with the internal representations of the patterns.”