FitNets: Hints for Thin Deep Nets
By Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio. 2014
4,770 citations recorded by the source corpus.
While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dacheng Tao
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Sergey Levine
- Percy Liang
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Yi Yang
- Jitendra Malik
- Bernt Schiele
In Model Architectures
- Yoshua Bengio
- Quoc V. Le
- Alan L. Yuille
- Dacheng Tao
- Kaiming He
- Xiaogang Wang
- Jiashi Feng
- Chunhua Shen
In Efficiency & Compression
Citations in the corpus
- Deep Residual Learning for Image Recognition (2015, methodology): “networks such as FitNet [35] and Highway [42] (Table 6), yet is among the state-of-the-art results (6.”
- Training Very Deep Networks (2015, methodology): “Deep supervision [24] has been shown to hurt performance of thin deep networks [25].”
- Highway Networks (2015, methodology): “We also show that architectures comparable to those recently presented by Romero et al. (2014) can be directly trained to obtain similar test set accuracy on the CIFAR-10 dataset without the need for a pre-trained teacher network.”
- TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “The most commonly used techniques include quantization (Gong et al., 2014), weights pruning (Han et al., 2015b), and knowledge distillation (KD) (Romero et al., 2014).”
- Sequence-Level Knowledge Distillation (2016, background): “Knowledge distillation approaches (Bucila et al., 2006; Ba and Caruana, 2014; Li et al., 2014; Hinton et al., 2015; Romero et al., 2015; Mou et al., 2015) learn a smaller student network to mimic the original teacher network by minimizing the loss (typically L2 or cross-entropy) between the student ”
- Wide Residual Networks (2016, background): “We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts.”
- Identity Mappings in Deep Residual Networks (2016, neutral): “re obtained with a mini-batch size of 128 except ywith a mini-batch size of 64 (code available at https://github.com/KaimingHe/resnet-1k-layers). CIFAR-10 error (%) NIN [15] 8.81 DSN [16] 8.22 FitNet [17] 8.39 Highway [7] 7.72 All-CNN [14] 7.25 ELU [12] 6.55 FitResNet, LSUV [18] 5.84 ResNet-110 [1] ”
- Learning by Distilling Context (2022, background): “There has been a large literature on distilling knowledge in a neural network (Hinton et al., 2015b; Adriana et al., 2015; Liu et al., 2019; Yang et al., 2020; Xie et al., 2020).”
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “…and substantial research efforts are invested in speeding up DNNs at run-time on both general-purpose (Vanhoucke et al., 2011; Gong et al., 2014; Romero et al., 2014; Han et al., 2015b) and specialized computer hardware (Farabet et al., 2011a,b; Pham et al., 2012; Chen et al., 2014a,b; Esser…”
- Born Again Neural Networks (2018, background): “In another recent work, Romero et al. (2014) aim to compress models by approximating the mappings between teacher and student hidden layers, us-ing linear projection layers to train the relatively narrower students.”
- Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning (2015, neutral): “A similar idea was also explored in the model compression case (Romero et al., 2015), where a deep and thin mimic network used a larger expert network’s intermediate features as guiding hints during training.”
- Learning without Forgetting (2016, background): “[19] builds on this work to transfer to a deeper network by applying extra guidance on the middle layer.”
- FractalNet: Ultra-Deep Neural Networks without Residuals (2016, background): “FitNets [27] explicitly couple students and teachers, forcing mimic behavior across several intermediate points in the network.”
- Patient Knowledge Distillation for BERT Model Compression (2019, background): “Romero et al. (2015) further demonstrated that intermediate representations learned by the large model can serve as hints to improve the training process and the final performance of the compact model.”
- All you need is a good init (2015, methodology): “Hinton et al. (2014) and Romero et al. (2015) take another approach to initialization and formulate training as mimicking teacher network predictions (so called knowledge distillation) and internal representations (so called Hints initialization) rather than minimizing the softmax loss.”