Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
By R.Monasson, R.Zecchina. 1995
We analytically derive the geometrical structure of the weight space in multilayer neural networks (MLN), in terms of the volumes of couplings associated to the internal representations of the training set. Focusing on the parity and committee machines, we deduce their learning and generalization capabilities both reinterpreting some known properties and finding new exact results. The relationship between our approach and information theory as well as the Mitchison–Durbin calculation is established. Our results are exact in the limit of a large number of hidden units, showing that MLN are a class of exactly solvable models with a simple interpretation of replica symmetry breaking.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Pratik Chaudhari
- Stefano Soatto
- Levent Sagun
- Alessandro Achille
- Yoshua Bengio
- Anna Choromanska
- Carlo Baldassi
- Charles H. Martin
Citations in the corpus
- Entropy-SGD: biasing gradient descent into wide valleys (2016, background): “, 2013). We consider the All-CNN-C network of Springenberg et al. (2014) with the only difference that a batch normalization layer is added after each convolutional layer; all other architecture and hyper-parameters are kept the same. We train for 200 epochs with SGD and Nesterov’s momentum during w”