BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

By Yeming Wen, Dustin Tran, Jimmy Ba. 2020

586 citations recorded by the source corpus.

Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the number of networks, which quickly becomes untenable. In this paper, we propose BatchEnsemble, an ensemble method whose computational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix to be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble is not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for a given mini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and out-of-distribution tasks, BatchEnsemble yields competitive accuracy and uncertainties as typical ensembles; the speedup at test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs. We further show that BatchEnsemble can easily scale up to lifelong learning on Split-ImageNet which involves 100 sequential learning tasks.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Meta & Continual Learning

Citations in the corpus

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “The LPHM layer can also be seen as leveraging “slow” weightsAi that are shared across adapters and capture general information and “fast” weightsBi that learn adapter-specific information for adaptation of each individual layer [25].”
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “Therefore we expect it to provide complementary benefits to approaches such as (efficient) ensembles and Bayesian neural networks [21, 42, 79] which marginalize over the representation parameters as well.”
Incremental Task Learning with Incremental Rank Updates (2022, methodology): “Dynamic network architectures [30,19,39,38,33,7,41] try to add new neurons to the model at additional new tasks, thus the performances on previous tasks are preserved by freezing the old parameters and only updating the newly added parameters.”
CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One (2022, neutral): “[47] proposed an efficient ensemble strategy to reduce extra parameter cost for task-specific sub-networks.”
FOSTER: Feature Boosting and Compression for Class-Incremental Learning (2022, background): “Many works [10,15,20,33,37] create new modules to handle the growing training distribution [26,41] dynamically.”
DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “Indeed, aiming for wider minima is particularly important in continual learning [14, 26].”
Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “…2018), generative replay-based (Mocanu et al., 2016; Shin et al., 2017; Liu et al., 2020; Ven et al., 2020), ensemble-based (Aljundi et al., 2017; Wen et al., 2020) and methods that select locally-redundant directions in the weight space (Kirkpatrick et al., 2017; Aljundi et al., 2018; Mallya et…”
Supermasks in Superposition (2020, methodology): “[51], SplitCIFAR100 randomly partitions CIFAR100 [24] into 20 different 5-way classification problems.”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, methodology): “Wen et al. (2020) apply ensembling on a rank-1 perturbation of each weight matrix and obtain strong empirical success without needing to learn the subspace.”
Analyzing the role of model uncertainty for electronic health records (2019, background): “Model uncertainty has made many methodological advances in recent years—including reparameterization-based variational Bayesian neural networks [3, 7, 19, 21], Monte Carlo dropout [8], deep ensembles and efficient alternatives [20, 33], and function priors [9, 13, 22].”
Lifelong Generative Modelling Using Dynamic Expansion Graph Model (2021, background): “Meanwhile, Batch Ensemble (BE) (Wen, Tran, and Ba 2020) is designed for classification tasks.”
Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “Plex (Tran et al., 2022) calls for BatchEnsemble layers (Wen et al., 2019) to be added in the model architecture during both pre-training and ﬁne-tuning.”
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks (2021, background): “If these models lie at the periphery of a wide and flat low loss region then ensembling them in their weight space (averaging), creates a model tending to the center of the region, which leads to performance improvement (Izmailov et al., 2019; Wen et al., 2020).”
Lifelong Mixture of Variational Autoencoders (2021, methodology): “Dataset L-MVAE-Dynamic BatchEnsemble [63]”
Deep Learning Through the Lens of Example Difficulty (2021, background): “…on the consensus-consistency score for all data points, irrespective of whether the 5Implementation details can be found in Appendix A.6.2 6Consensus-consistency score is a measure of uncertainty and can be used for calibration (Lakshminarayanan et al., 2017; Wenzel et al., 2020; Wen et al., 2019).”
In the Wild: From ML Models to Pragmatic ML Systems (2020, methodology): “Some recent continual learning works have used Split-ImageNet [47].”
No One Representation to Rule Them All: Overlapping Features of Training Methods (2021, background): “It is widely understood that good ensembles are made of models that are both accurate and make independent errors (Perrone & Cooper, 1992; Opitz & Maclin, 1999; Wen et al., 2020).”
Lifelong Infinite Mixture Model Based on Knowledge-Driven Dirichlet Process (2021, methodology): “Furthermore, by accumulating knowledge, while enabling fast inference across data domains, using a lightweight model is an attractive feature in LLL, which does not appear in existing lifelong mixtures or ensemble models [20, 47].”