Neural networks with late-phase weights
By Johannes von Oswald, Seijin Kobayashi, João Sacramento, Alexander Meulemans, Christian Henning, Benjamin F. Grewe. 2020
40 citations recorded by the source corpus.
The largely successful method of training neural networks is to learn their weights using some variant of stochastic gradient descent (SGD). Here, we show that the solutions found by SGD can be further improved by ensembling a subset of the weights in late stages of learning. At the end of learning, we obtain back a single model by taking a spatial average in weight space. To avoid incurring increased computational costs, we investigate a family of low-dimensional late-phase weight models which interact multiplicatively with the remaining parameters. Our results show that augmenting standard models with late-phase weights improves generalization in established benchmarks such as CIFAR-10/100, ImageNet and enwik8. These findings are complemented with a theoretical analysis of a noisy quadratic problem which provides a simplified picture of the late phases of neural network learning.
Subfields: Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Optimization & Training
- Maria R. Cervera
- Francesco D'Angelo
- Ali Farhadi
- Johannes von Oswald
- Carlos Guestrin
- Benjamin Ehret
- Regina Traber
- Mohammad Rastegari
Citations in the corpus
- Posterior Meta-Replay for Continual Learning (2021, methodology): “Despite the fact that we could improve upon previously reported results for the PR-DIRAC baseline in MLP-400,400 (termed HNET+ENT in von Oswald et al. (2020)), this deterministic solution generally performs worse at inferring task identity than probabilistic approaches, notably PR-BBB. Interestingly”
- Learning Neural Network Subspaces (2021, background): “We highlight a few recent observations which have advanced understanding of the neural network optimization landscape (Dauphin et al., 2014; Li et al., 2018a;b; Fort & Jastrzebski, 2019; Evci et al., 2019; Frankle, 2020; Oswald et al., 2021).”