Deep Ensembles: A Loss Landscape Perspective

By Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan. 2019

770 citations recorded by the source corpus.

Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “Various notions of underspecification are well-documented in the ML literature; for example, it is a core idea in deep ensembles, double descent, Bayesian deep learning, and loss landscape analysis (Lakshminarayanan et al., 2017; Fort et al., 2019; Belkin et al., 2018; Nakkiran et al., 2020).”
Linear Mode Connectivity in Multitask and Continual Learning (2020, background): “We chose this slightly non-conventional setup to minimize the potential number of confounding factors that lead to discrepancies be- tween the two solutions (Fort et al., 2019).”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, methodology): “The diversity of predictions returned by different members of an ensemble is an important indicator of the quality of uncertainty quantification (Fort et al., 2019) and of the robustness of the ensemble (Pang et al.”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, methodology): “In this section, we provide visualization of some commonly used diversity metrics proposed in Fort et al. (2019).”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “Generally, such approaches ensemble models trained from scratch; while ensembles of training checkpoints lose some diversity (Fort et al., 2019), they offer a cheaper alternative capturing some of the beneﬁts (Chen et al., 2017a).”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “[8] considered the diversity of predictions produced by models from a single SGD run, and models from independent SGD runs, and suggested to ensemble averages of SGD iterates.”
Adversarial vulnerability of powerful near out-of-distribution detection (2022, background): “A loss landscape approach to that is discussed in [23].”
Deep Ensembles for Low-Data Transfer Learning (2020, background): “The source of diversity for deep networks has been studied (Fort et al., 2019; Wenzel et al., 2020), though not thoroughly in the low-data regime.”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, methodology): “Moreover, as diversity metric, we take the predictive disagreement of the ensemble members normalized by (1-accuracy), as used in [22].”
Learning Neural Network Subspaces (2021, methodology): “Fort et al. (2019) show that independently trained models have weight vectors with a cosine similarity of approximately 0, unlike models with a shared trajectory.”
Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains (2020, methodology): “A model which has been generated by transfer learning, and the original "base model" have a close relationship [4] and it has been shown that linearly interpolating between the weights of two such generative models produces outputs which are approximately an interpolation of the two learned domains…”
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, background): “Related works (Frankle et al., 2020; Fort et al., 2019) show that simply averaging two SGD solutions would fail.”
Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win (2020, methodology): “As suggested by the analysis of Fort, Hu, and Lakshminarayanan (2020), ensembles of different solutions are more robust, and generalize better, than ensembles of similar solutions.”
Neural networks with late-phase weights (2020, background): “While mode coverage is usually a sought after property when learning neural network ensembles (Fort et al., 2020), here it would preclude us from taking the averaged model at the end of learning (Eq.”
Editing Models with Task Arithmetic (2022, neutral): “[25] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.”
FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling (2023, background): “While existing work has focused on improvements towards aggregate performance (Fort et al., 2019; Gupta et al., 2022; Opitz & Maclin, 1999) or gains in efficiency over a single larger model (Wang et al., 2020; Worts-man et al., 2022), there has been limited consideration of how sensitive ensembling…”
Efficient Continual Learning Ensembles in Neural Network Subspaces (2022, methodology): “Inspired by bootstrapping (Breiman, 1996), deep ensembles initialize and train multiple neural networks independently (Lakshminarayanan et al., 2017; Fort et al., 2019).”