Training Deep Neural Networks in Generations: A More Tolerant Teacher Educates Better Students

By Chenglin Yang, Lingxi Xie, Siyuan Qiao, Alan L. Yuille. 2018

149 citations recorded by the source corpus.

We focus on the problem of training a deep neural network in generations. The flowchart is that, in order to optimize the target network (student), another network (teacher) with the same architecture is first trained, and used to provide part of supervision signals in the next stage. While this strategy leads to a higher accuracy, many aspects (e.g., why teacher-student optimization helps) still need further explorations.This paper studies this problem from a perspective of controlling the strictness in training the teacher network. Existing approaches mostly used a hard distribution (e.g., one-hot vectors) in training, leading to a strict teacher which itself has a high accuracy, but we argue that the teacher needs to be more tolerant, although this often implies a lower accuracy. The implementation is very easy, with merely an extra loss term added to the teacher network, facilitating a few secondary classes to emerge and complement to the primary class. Consequently, the teacher provides a milder supervision signal (a less peaked distribution), and makes it possible for the student to learn from inter-class similarity and potentially lower the risk of over-fitting. Experiments are performed on standard image classification tasks (CIFAR100 and ILSVRC2012). Although the teacher network behaves less powerful, the students show a persistent ability growth and eventually achieve higher classification accuracies than other competitors. Model ensemble and transfer feature extraction also verify the effectiveness of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Qi Tian
3. Jianping Gou
4. S. Maybank
5. Baosheng Yu
6. Lin Wang
7. Kuk-Jin Yoon
8. Lingxi Xie

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Qi Tian
3. Yi Yang
4. Lingxi Xie
5. Luc Van Gool
6. Alan L. Yuille
7. Ming-Hsuan Yang
8. Xiaogang Wang

Citations in the corpus

• Knowledge Distillation: A Survey (2020, methodology): “Snapshot distillation (Yang et al. 2019b) is a special variant of self-distillation, in which knowledge in the earlier epochs of the network (teacher) is transferred into its later epochs (student) to support a supervised training process within the same network.”
• Circumventing Outliers of AutoAugment with Knowledge Distillation (2020, background): “Beyond its wide application on model compression (large teacher, small student [33,16]) and model initialization (small teacher, large student [34,3]), researchers later proposed to use it for standard network training, with the teacher and student models sharing the same network architecture [9,50,”
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, background): “[246] initiate to put strictness on the teacher while Yu et al.”
• Self-Distillation as Instance-Specific Label Smoothing (2020, background): “ter performance is the increasing uncertainty and diversity in teacher predictions. Similar observations that more “tolerant” teacher predictions lead to better students were also made by Yang et al. [27]. Indeed, due the monotonicity and convexity of the negative log likelihood function, since the ”
• Creating Something From Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing (2020, background): “Recently, many efforts were made to study what the dark knowledge is [41, 40], and/or how to efficiently take advantage of such knowledge [11, 42, 35, 2].”
• RomeBERT: Robust Training of Multi-Exit BERT (2021, methodology): “It has been widely applied in CNN based models (Furlanello et al., 2018; Xie et al., 2020; Yang et al., 2019).”
• Decoupled Certainty-Driven Consistency Loss for Semi-supervised Learning (2019, background): “From the view of knowledge distillation,learning true inter-class similarity information on each image can help model to generalize well [7],however, that information is Figure 2: Our proposed circle-shape framework with 2 decoupled students and teachers.”