Self-Distillation Amplifies Regularization in Hilbert Space

By Hossein Mobahi, Mehrdad Farajtabar, Peter L. Bartlett. 2020

286 citations recorded by the source corpus.

Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another. In particular, when the architectures are identical, this is called self-distillation. The idea is to feed in predictions of the trained model as new target values for retraining (and iterate this loop possibly a few times). It has been empirically observed that the self-distilled model often achieves higher accuracy on held out data. Why this happens, however, has been a mystery: the self-distillation dynamics does not receive any new information about the task and solely evolves by looping over training. To the best of our knowledge, there is no rigorous understanding of why this happens. This work provides the first theoretical analysis of self-distillation. We focus on fitting a nonlinear function to training data, where the model space is Hilbert space and fitting is subject to L2 regularization in this function space. We show that self-distillation iterations modify regularization by progressively limiting the number of basis functions that can be used to represent the solution. This implies (as we also verify empirically) that while a few rounds of self-distillation may reduce over-fitting, further rounds may lead to under-fitting and thus worse performance.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, background): “Another two related works are [30] which provides theoretical analysis of self-distillation and CRD [50] which shows distillation improves the transferability across datasets.”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “Numerous variants of KD have been proposed, including feature distillation [28, 73], iterative distillation [43, 67], and self-distillation [65, 44, 18, 29].”
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting (2021, background): “For example, self-distillation uses trained model’s outputs for the subsequent training and plays an interesting role of regularization (Mobahi et al., 2020).”
Knowledge Distillation: A Survey (2020, methodology): “Recently, self-distillation has been theoretically analyzed in Mobahi et al. (2020), and its improved performance experimentally demonstrated in Zhang and Sabuncu (2020).”
Dropout as an Implicit Gating Mechanism For Continual Learning (2020, background): “The idea of using knowledge distillation [11, 20, 21] is also found to be a successful regularizer in several works [15, 14].”
Essentials for Class Incremental Learning (2021, background): “Self-distillation [7, 20] is a form of knowledge distillation in which the teacher and student networks have the same architecture.”
Transferring Inductive Biases through Knowledge Distillation (2020, background): “This could be due to the regularization effect of distillation [27, 36].”
R-Drop: Regularized Dropout for Neural Networks (2021, background): “In our setting, the teacher and student are the dropout instantiations of the same model, thus it resembles self-knowledge distillation [44] scenario.”
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning (2020, background): “• (Self-distillation [36, 63] also improves test accuracy, see Figure 1): if we further train (using GD from random initialization) another network f ′ to match the output of a single model f1 merely over the same training data set Z, then f ′ can be trained efficiently and w.”
Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (2020, background): “[16] shows that self-distillation has a regularization effect on student’s logits.”
Localization Distillation for Object Detection (2022, background): “For the reason why self-distillation can facilitate the model accuracy, [34] firstly unveils the mystery by theoretically analyzing self-distillation in Hilbert Space.”
Does Knowledge Distillation Really Work? (2021, background): “Indeed, it is becoming well-known that in self-distillation the student fails to match the teacher and, paradoxically, student generalization improves as a result [13, 32].”
A statistical perspective on distillation (2021, background): “But what are the formal statistical benefits of using soft over one-hot labels, and what is the “ideal” set of soft labels? While several works have studied various aspects of distillation (Lopez-Paz et al., 2016; Phuong & Lampert, 2019; Mobahi et al., 2020; Ji & Zhu, 2020; Zhang & Sabuncu, 2020; Zh”
Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation (2021, background): “In the case of self-distillation (SD), where the teacher model and the student model are the same, such an approach progressively restricts the number of basis functions to represent the solution [Mobahi et al. , 2020].”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, result): “This improvement has been attributed to the ease of predicting the output distribution of the teacher compared to the original ‘hard’ labels (Mobahi et al., 2020; Gotmare et al., 2019).”
Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020, background): “We suspect that imitation models can sometimes outperform their victims because distillation can act as a regularizer (Furlanello et al., 2018; Mobahi et al., 2020).”