Ensemble Distribution Distillation

By Andrey Malinin, Bruno Mlodozeniec, Mark John Francis Gales. 2019

278 citations recorded by the source corpus.

Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different \emph{forms} of uncertainty. However, ensembles come at a computational and memory cost which may be prohibitive for many applications. There has been significant work done on the distillation of an ensemble into a single model. Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble. However, information about the \emph{diversity} of the ensemble, which can yield estimates of different forms of uncertainty, is lost. This work considers the novel task of \emph{Ensemble Distribution Distillation} (EnD$^2$) --- distilling the distribution of the predictions from an ensemble, rather than just the average prediction, into a single model. EnD$^2$ enables a single model to retain both the improved classification performance of ensemble distillation as well as information about the diversity of the ensemble, which is useful for uncertainty estimation. A solution for EnD$^2$ based on Prior Networks, a class of models which allow a single neural network to explicitly model a distribution over output distributions, is proposed in this work. The properties of EnD$^2$ are investigated on both an artificial dataset, and on the CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that EnD$^2$ can approach the classification performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassification and out-of-distribution input detection.

Subfields: Artificial Intelligence, Control and Systems Engineering, Generative Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Generative Models

In Efficiency & Compression

Citations in the corpus

Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, background): “Ensemble models on the other hand, have been shown to be promising teachers from the early work of [9] until recent works in various domains [10, 68, 54, 56] and with techniques to boost the their performance [52, 17, 41].”
Robust fine-tuning of zero-shot models (2021, background): “through distillation into a single model [65, 94]).”
Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance (2021, background): “(2), as follows (Malinin, 2019): .”
Does Knowledge Distillation Really Work? (2021, methodology): “[28] proposed a method to model the implicit distribution over predictive distributions from which the ensemble component predictive distributions are drawn, rather than just the ensemble model average.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “[151] point out that estimating the model’s uncertainty is crucial since it ensures more reliable knowledge to be transferred.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “Ensembles involve memory and computational cost which is not acceptable in many application [197].”
Uncertainty Estimation in Autoregressive Structured Prediction (2021, methodology): “Rejection curves are summarised using the Prediction Rejection Ratio (PRR) (Malinin, 2019; Malinin et al., 2020), describe in appendix D.”
Self-Distillation as Instance-Specific Label Smoothing (2020, background): “More recently, there has also been work on the importance of distillation from an ensemble of model [24], which provides a complementary view on the role of predictive diversity.”
Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution (2020, methodology): “KD has been used to provide a fast approximation to Bayesian Neural Networks [4, 38, 16].”
Complementary Relation Contrastive Distillation (2021, background): “Knowledge distillation aims to transfer the knowledge from one deep learning model (the teacher) to another (the student), such as distilling a large network into a smaller one [19, 49, 2, 48, 12] or ensembling a collection of models into a single model [29, 37, 27, 45].”
A survey of uncertainty in deep neural networks (2021, methodology): “instead of directly predicting the output [90], [45].”
Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data (2020, background): “This problem has been studied for many years [12] and has been discussed in several views such as rejection [8, 5], anomaly detection [1], open set recognition [2], and uncertainty estimation [22, 23, 24].”
Trusted Multi-View Classification With Dynamic Evidential Fusion (2022, methodology): “Typical examples of Dirichlet distribution [69], [82] and subjective opinion.”
Hydra: Preserving Ensemble Diversity for Model Distillation (2020, methodology): “One way to quantify uncertainty is through model uncertainty (Depeweg et al., 2017; Malinin et al., 2019) which measures the spread or disagreement of an ensemble.”