Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

By Zeyuan Allen-Zhu, Yuanzhi Li. 2020

476 citations recorded by the source corpus.

We formally study how Ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using Knowledge Distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We empirically show that ensemble/knowledge distillation in deep learning works very differently from traditional learning theory, especially differently from ensemble of random feature mappings or the neural-tangent-kernel feature mappings, and is potentially out of the scope of existing theorems. Thus, to properly understand ensemble and knowledge distillation in deep learning, we develop a theory showing that when data has a structure we refer to as "multi-view", then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the "dark knowledge" is hidden in the outputs of the ensemble -- that can be used in knowledge distillation -- comparing to the true data labels. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “• Knowledge distillation An ensemble of deep learning models can typically improve the performance, and this superior performance can be distilled into a single model using knowledge distillation [2].”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “Knowledge distillation [2], where knowledge present in an ensemble of models is distilled into a single model, has been borrowed to the context of prompt combination by Schick and Schütze [165, 166, 167] and by Gao et al.”
A Survey on Knowledge Distillation of Large Language Models (2024, unknown): “(Allen-Zhu and Li, 2020; Wang et al., 2022a; Sun et al., 2024b; Yang et al., 2024a; Jung et al., 2023; Huang et al., 2023a; Gulcehre et al., 2023; Yuan et al., 2024a; Xu et al., 2023b; Zelikman et al., 2022; Chen et al., 2024b; Zheng et al., 2024; Li et al., 2024c; Zhao et al., 2024; Singh et al.,…”
R-Drop: Regularized Dropout for Neural Networks (2021, background): “Minimizing the KL-divergence between the output distributions of two different models correlates with knowledge distillation [23, 15, 2, 36, 13, 78], where the two models refer to teacher and student, respectively.”
TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023, methodology): “Several methods have been proposed to compress or distill large LMs into smaller ones, such as knowledge distillation [10, 2], pruning [8], and quantization [13].”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, result): “Consistent with prior findings on self-distillation (Pham et al., 2022; Allen-Zhu and Li, 2020), this simple process significantly improves the performance of our task model, as confirmed by our ablation study (§4.5).”