Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

By Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, Andrew Gordon Wilson. 2018

951 citations recorded by the source corpus.

The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “One possible explanation of this immediate impact is the presence of no energy valleys in deep learning loss landscape between minima of separately trained models [18, 24].”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “Current efforts to design more effective methods for exploring loss landscapes (Fort et al., 2019; Garipov et al., 2018) could play an important role here, and there are opportunities to import ideas from the sensitivity analysis and partial identification subfields in causal inference and inverse p”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Most applications study models along the same optimization trajectory, e.g. (Ruppert, 1988; Polyak, 1990; Szegedy et al., 2016; Izmailov et al., 2018; Zhang et al., 2019; Kaddour et al., 2022; Junczys-Dowmunt et al., 2016).”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “, integrating multiple minima in parameter or function space along the low-error path, since connecting them ensures flatness on that path [59], [129], [182], [308], [430], [450].”
A Simple Baseline for Bayesian Uncertainty in Deep Learning (2019, background): “However, we would expect that even in the infinite data limit the posterior of DNNs would converge to something very non-Gaussian, with connected modes surrounded by gorges of zero posterior density [17].”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, methodology): “Building upon this insight, Garipov et al. [2018] also developed Fast Geometric Ensembling (FGE) to sample multiple nearby points in weight space to create high performing ensembles in the time required to train a single DNN. FGE uses a high frequency cyclical learning rate with SGD to select networ”
Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector (2021, neutral): “Observation 1 (Connectivity) [10,14] There exists a continuous path between minima of neural network architectures, where each point along this path has a low loss.”
SWAD: Domain Generalization by Seeking Flat Minima (2021, methodology): “Following Garipov et al. [24], we choose three model weights θ 1 , θ 2 , θ 3 and deﬁne two dimensional weight plane from the weights: where ˆ u = u/ (cid:107) u (cid:107) and ˆ v = v/ (cid:107) v (cid:107) are orthonormal bases of the weight plane.”
Linear Mode Connectivity in Multitask and Continual Learning (2020, methodology): “…inspired by recent work on mode con- ∗Equal contribution 1The code is available at:https://github.com/imirzadeh/MC-SGD nectivity (Draxler et al., 2018; Garipov et al., 2018; Frankle et al., 2020) finding that different optima obtained by gradient-based optimization methods are connected by simple…”
What is being transferred in transfer learning? (2020, background): “It has been observed in the literature that any two minimizers of a deep network can be connected via a non-linear low-loss path [11, 9, 10].”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, background): “Garipov et al. (2018) proposed fast geometric ensemble where it ﬁnds modes that can be connected by simple curves, and each mode can taken as one ensemble member.”
Essentially No Barriers in Neural Network Energy Landscape (2018, methodology): “After submission of this work to the International Machine Learning Conference (ICML) 2018, (Garipov et al., 2018) independently reported that they also constructed paths between neural network minima.”
Lookahead Optimizer: k steps forward, 1 step back (2019, background): “[8] observe that the final weights of two independently trained neural networks can be connected by a curve with low loss.”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “Draxler et al. (2018) and Garipov et al. (2018) show that the modes of standard vision networks trained from different initializations are connected by piece-wise linear paths of constant error or loss.”
Rigging the Lottery: Making All Tickets Winners (2019, background): “However, this is only one of inﬁnitely many paths between the two points (Garipov et al., 2018; Draxler et al., 2018) and does not imply the nonex-istence of such a path.”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “using cyclic learning rate as done in fast geometric ensembling [Garipov et al., 2018] can yield further benefits.”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “Not only are the likelihoods diffuse, but different settings of the parameters correspond to a diverse variety of compelling hypotheses for the data (Garipov et al., 2018; Izmailov et al., 2019).”
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning (2020, methodology): “, snapshot ensembles [10] and fast geometric ensembles [6]), or obtained from a sampling process (e.”
Large Scale Structure of Neural Network Loss Landscapes (2019, methodology): “One of the first studies on curvature were carried out by LeCun et al. [1998], Sagun et al. [2016]. A significant, though not well understood, phenomenon is that curvature correlates with generalization in many networks.”
Explaining Knowledge Distillation by Quantifying the Knowledge (2020, positive): “We utilize ∑m k=1 ‖wk−wk−1‖ ‖w0‖ , named “weight distance” , to measure the learning effect at m-th epoch [12, 7].”