Distilling the Knowledge in a Neural Network
By Geoffrey E. Hinton, Oriol Vinyals, Joseph Dean. arXiv.org, 2015
25,160 citations recorded by the source corpus.
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
Subfields: Artificial Intelligence, Efficiency & Compression, Signal Processing
View Paper: https://arxiv.org/abs/1503.02531
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Sergey Levine
- Luke S. Zettlemoyer
- Dacheng Tao
In Efficiency & Compression
In Signal Processing
Citations in the corpus
- Xception: Deep Learning with Depthwise Separable Convolutions (2016, background): “Since its first introduction, Inception has been one of the best performing family of models on the ImageNet dataset [14], as well as internal datasets in use at Google, in particular JFT [5].”
- Using the Output Embedding to Improve Language Models (2016, background): “Independently and concurrently with our work (Inan et al., 2016) presented an explanation for weight tying in NNLMs based on (Hinton et al., 2015).”
- Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “Many recent approaches (including the consisten-tency regularization methods discussed below and our own method) train the student with soft targets from the teacher’s output distribution rather than a hard label, making the procedure more akin to knowledge distillation (Hinton et al., 2015).”
- FitNets: Hints for Thin Deep Nets (2014, methodology): “In order to obtain a faster inference, we explore the recently proposed compression framework (Hinton & Dean, 2014), which trains a student network , from the softened output of an ensemble of wider networks, teacher network .”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “Some current work along these lines include distillation [Hinton et al., 2015; Sanh et al., 2019; Jiao et al., 2019], parameter sharing [Lan et al.”
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “To overcome this problem, we resort to the following strategy, which closely resembles knowledge distillation (Hinton et al., 2015).”
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019, methodology): “Following Hinton et al. [2015] we used a softmax-temperature : where T controls the smoothness of the output distribution and z i is the model score for the class i .”
- TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “In this paper we focus on knowledge distillation, an idea proposed by Hinton et al. (2015) in a teacher-student framework.”
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “(2019b) demonstrate that knowledge distillation (Hinton et al., 2015; Furlanello et al., 2018) can lead to student networks that outperform their teachers.”
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, methodology): “…a process of distilling or transferring the knowledge from a (set of) large, cumbersome model(s) to a lighter, easier-to-deploy single model, without significant loss in performance (Bucilu et al., 2006; Hinton et al., 2015; Balan et al., 2015; Ba et al., 2016; Chen et al., 2015; Tan et al., 2019).”
- Sequence-Level Knowledge Distillation (2016, methodology): “similarity between classes) and has less variance in gradients (Hinton et al., 2015).”
- KERMIT: Generative Insertion-Based Modeling for Sequences (2019, methodology): “We also follow prior work (Gu et al., 2018; Stern et al., 2018, 2019; Lee et al., 2018) in using distillation (Hinton et al., 2015; Kim and Rush, 2016) to train our models.”
- BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, methodology): “Intuitively, distillation is effective because the teacher’s output distribution over classes provides more training signal than a one-hot label; Hinton et al. (2015) suggest that teacher outputs contain “dark knowledge” capturing additional information about training examples.”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “In this section we explore whether it is possible to blend the benefits of dense and sparse models via knowledge distillation (Hinton et al., 2015).”
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “…problem of making transformers efficient have also been tackled from an extensive number of angles such as (but not limited to) distillation (Hinton et al., 2015), compression (Zafrir et al., 2019), parameter sharing (Lan et al., 2019; Tay et al., 2019; Zhang et al., 2021), efficient…”
- Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “Our method is inspired by knowledge distillation (Hinton et al., 2015), and uses the reader model to obtain synthetic labels to train the retriever model.”