Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

By Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, Dhruv Batra. 2015

343 citations recorded by the source corpus.

Convolutional Neural Networks have achieved state-of-the-art performance on a wide range of tasks. Most benchmarks are led by ensembles of these powerful learners, but ensembling is typically treated as a post-hoc procedure implemented by averaging independently trained models with model variation induced by bagging or random initialization. In this paper, we rigorously treat ensembling as a first-class problem to explicitly address the question: what are the best strategies to create an ensemble? We first compare a large number of ensembling strategies, and then propose and evaluate novel strategies, such as parameter sharing (through a new family of models we call TreeNets) as well as training under ensemble-aware and diversity-encouraging losses. We demonstrate that TreeNets can improve ensemble performance and that diverse ensembles can be trained end-to-end under a unified loss, achieving significantly higher "oracle" accuracies than classical ensembles.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors (2016, methodology): “However, in order to take advantage of models with complementary strengths, we also explicitly encourage diversity by pruning away models that are too similar to previously selected models (c.f., [18]).”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, methodology): “[36] independently observed that training on entire dataset with random initialization was better than bagging for deep ensembles, however their goal was to improve (3)Not to be confused with Generative Adversarial Networks (GANs).”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “However, works such as TreeNets [153] (Section 6.”
Deep Ensembles: A Loss Landscape Perspective (2019, background): “However, it has been empirically observed by Lakshminarayanan et al. (2017); Lee et al. (2015) that training individual networks with just random initialization is sufﬁcient in practice and using the bootstrap even hurts performance in some cases (e.g. for small ensemble sizes).”
No New-Net (2018, background): “Since the label deﬁnitions between the BraTS dataset and our own diﬀer slightly, we add a second segmentation layer (1x1x1 convolution) at the end, which acts as a supervised version of m heads [26].”
Deep Ensembles for Low-Data Transfer Learning (2020, methodology): “ms of pre-training. This 8 approach of combining different pre-trained models is highly complimentary with efforts which train ensembles end-to-end with diversity-encouraging losses, such as those in Lee et al. (2015) and Webb et al. (2019). Lastly, works such as Batch Ensembles (Wen et al., 2020) a”
Neural networks with late-phase weights (2020, methodology): “Such gradient accumulation has been previously used when learning ensembles (Lee et al., 2015; Wen et al., 2020) and multi-task models (Rebuffi et al.”
Improving Confidence Estimates for Unfamiliar Examples (2018, methodology): “Multi-head deep networks [24, 31] emulate ensembles and are shown to outperform MC-dropout.”
Objects are Different: Flexible Monocular 3D Object Detection (2021, background): “Ensemble learning [1, 12, 17, 23, 39] generates multiple models strategically and combines their predictions for better performance.”
A survey of uncertainty in deep neural networks (2021, background): “[230], who found bagging to worsen the predictive accuracy of ensemble methods on the investigated tasks.”
Communication Quantization for Data-Parallel Training of Deep Neural Networks (2016, background): “A Caffe variant, mpi-caffe [8], does support distributed computation, but does no quantization and has little in-depth evaluation of scalability.”
High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach (2018, background): “Recent work reported that parameter resampling offered superior performance to both bootstrap resampling, and a combination of the two (Lee et al., 2015; Lakshminarayanan et al., 2017).”
Quality Resilient Deep Neural Networks (2017, background): “Inverted-TreeNets TreeNets, as described in [13], only share the lower layers of a network, because the lower layers are assumed to describe similar features.”
On building ensembles of stacked denoising auto-encoding classifiers and their further improvement (2018, methodology): “From our point of view, the studies carried out in [61] are specially relevant.”
Neural Ensemble Search for Performant and Calibrated Predictions (2020, background): “We focus on ensembles of independently trained base learners, as this is a simple approach leading to strong ensembles (see [34] for a comparison of ensembling approaches).”
Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System (2019, methodology): “del generalize over a highly skewed distribution of relations. Pentina and Lampert [22] studied a variant of multi-task learning in which annotated data was available on some of the tasks. Lee et al. [17] studied the performance of different ensemble methods under the framework of multi-task learnin”