Hyperparameter Ensembles for Robustness and Uncertainty Quantification

By Florian Wenzel, Jasper Snoek, Dustin Tran, Rodolphe Jenatton. 2020

256 citations recorded by the source corpus.

Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration. The recently introduced batch ensembles provide a drop-in replacement that is more parameter efficient. In this paper, we design ensembles not only over weights, but over hyperparameters to improve the state of the art in both settings. For best performance independent of budget, we propose hyper-deep ensembles, a simple procedure that involves a random search over different hyperparameters, themselves stratified across multiple random initializations. Its strong performance highlights the benefit of combining models with both weight and hyperparameter diversity. We further propose a parameter efficient version, hyper-batch ensembles, which builds on the layer structure of batch ensembles and self-tuning networks. The computational and memory costs of our method are notably lower than typical ensembles. On image classification tasks, with MLP, LeNet, and Wide ResNet 28-10 architectures, our methodology improves upon both deep and batch ensembles.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “We conjecture that many combinations of these choices would reveal a far larger validation-equivalent set of high-performing predictors F∗, a conjecture that has been partially borne out by concurrent work (Wenzel et al., 2020).”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “, 2020), including greedy selection strategies (Caruana et al., 2004; 2006; Lévesque et al., 2016; Wenzel et al., 2020).”
Deep Learning Through the Lens of Example Difficulty (2021, background): “…on the consensus-consistency score for all data points, irrespective of whether the 5Implementation details can be found in Appendix A.6.2 6Consensus-consistency score is a measure of uncertainty and can be used for calibration (Lakshminarayanan et al., 2017; Wenzel et al., 2020; Wen et al., 2019).”
No One Representation to Rule Them All: Overlapping Features of Training Methods (2021, background): “More relevant to us, Wenzel et al. (2020) and Zaidi et al. (2021) have explored the effect of random hyper-parameters and architectures respectively, ﬁnding best ensembles when combining diverse models, albeit still considering similar frameworks.”
Laplace Redux - Effortless Bayesian Deep Learning (2021, methodology): “[98] Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton.”
Revisiting the Calibration of Modern Neural Networks (2021, methodology): “In fact, over the last few years, there have been many reports that calibration of modern neural networks can be surprisingly poor, despite the advances in accuracy (e.g. Guo et al. 2017; Lakshminarayanan et al. 2017; Malinin & Gales 2018; Thulasidasan et al. 2019; Hendrycks et al. 2020b; Ovadia et ”
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022, background): “Ensembling over multiple architectures and hyperparameter settings can yield a rough approximation to a distribution over these hyperparameters and has been shown to improve performance (Zaidi et al., 2021; Wenzel et al., 2020).”
Deep Isolation Forest for Anomaly Detection (2022, background): “, [23] uses both weight and hyperparameter diversity, [24] uses repulsive terms to ensure individual diversity, [25] increases diversity by reducing spurious correlations among features by adversarially preventing features from being conditionally predictable from each other, and [26] proposes a dis”