Snapshot Ensembles: Train 1, get M for free

By Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, Kilian Q. Weinberger. 2017

1,092 citations recorded by the source corpus.

Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost. We achieve this goal by training a single neural network, converging to several local minima along its optimization path and saving the model parameters. To obtain repeated rapid convergence, we leverage recent work on cyclic learning rate schedules. The resulting technique, which we refer to as Snapshot Ensembling, is simple, yet surprisingly effective. We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Fixing Weight Decay Regularization in Adam (2017, methodology): “Nevertheless, state-of-the-art results for popular image classiﬁcation datasets, such as CIFAR-10 and CIFAR-100 Krizhevsky (2009), are still obtained by applying SGD with momentum (Huang et al., 2016; 2017; Loshchilov & Hutter, 2016; Gastaldi, 2017).”
Born Again Neural Networks (2018, methodology): “(3) Finally, similarly to ensembling multiple snapshots (Huang et al., 2017) of SGD with restart (Loshchilov & Hutter, 2016), we produce Born-Again Network Ensembles (BANE) by averaging the prediction of multiple generations of BANs. f̂k(x) = k∑ i=1 f(x, θi)/k. (4) We find the improvements of the…”
Don't Use Large Mini-Batches, Use Local SGD (2018, background): “Furthermore, recent work by (Loshchilov & Hutter, 2016; Huang et al., 2017) leverages cyclic learning rate schedules either improving the anytime performance of deep neural network training, or ensembling multiple neural networks at no additional training cost.”
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs (2018, methodology): “In particular, Snapshot Ensembles adopt cyclical learning rates with cycle length on the scale of 20 to 40 epochs from the beginning of the training as they are trying to do large steps in the weight space.”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, background): “Ensembles of neural networks beneﬁt from the above observation to achieve better performance by averaging or majority voting on the output of each ensemble member (Xie et al., 2013; Huang et al., 2017).”
An Alternative View: When Does SGD Escape Local Minima? (2018, background): “It is also observed that the correct learning rate schedule (small or large) is crucial for escaping bad local minima [15, 21].”
Simple And Efficient Architecture Search for Convolutional Neural Networks (2017, methodology): “For further improving the results, we take snapshots of the best models from every iteration while running our algorithm following the idea of Huang et al. (2017) when using SGDR (Loshchilov & Hutter, 2017) for training.”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, methodology): “It is a strategy [26, 85] to use multiple algorithms to improve machine learning performance, and which is proved to be effective to reduce the problems related to overfitting [35, 68].”
LSTM Fully Convolutional Networks for Time Series Classification (2017, background): “[32], multiple local minima lie along the optimization path of amodel.”
Self-Referenced Deep Learning (2018, methodology): “On CIFAR datasets, the training budget is 300 epochs for DenseNet, and 200 epochs for ResNet and Wide ResNet models, same as [16].”
YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors (2022, methodology): “Model re-parametrization techniques [71, 31, 75, 19, 33, 11, 4, 24, 13, 12, 10, 29, 14, 78] merge multiple computational modules into one at inference stage.”
Using Mode Connectivity for Loss Landscape Analysis (2018, methodology): “Huang et al. (2017) propose an ensembling strategy using the set of iterates before restarts and claim that, when using the learning rate annealing cycles, the optimization path converges to and escapes from several local minima.”
Strong Baselines for Neural Semi-Supervised Learning under Domain Shift (2018, background): “In general, the line be-tween “explicit” and “implicit” ensembling (Huang et al., 2017), like dropout (Srivastava et al., 2014) or temporal ensembling (Saito et al., 2017), is more fuzzy.”