Deep Mutual Learning

By Ying Zhang, Tao A. Xiang, Timothy M. Hospedales, Huchuan Lu. 2017

1,998 citations recorded by the source corpus.

Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, in order to meet the low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy. Different from the one-way transfer between a static pre-defined teacher and a student in model distillation, with DML, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on both category and instance recognition tasks. Surprisingly, it is revealed that no prior powerful teacher network is necessary - mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “While most methods focus on distilling knowledge from a pre-trained teacher model [28, 29, 30, 31, 32], online distillation [33, 34] simultaneously trains multiple models and use their ensemble as the teacher.”
Group Knowledge Transfer: Collaborative Training of Large CNNs on the Edge (2020, background): “Previous works only consider transferring knowledge from a large network to a smaller one [20, 21, 22, 23], or they transfer knowledge from a group, but each member in the group shares the same large model architecture or a large portion of the neural architecture with specific tail or head layers [”
Contrastive Representation Distillation (2019, background): “In (Zhang et al., 2018b), a Deep Mutual Learning setting was proposed where the teacher and student networks are trained simultaneously rather than sequentially.”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “Also, studies such as DML (Zhang et al. 2017) and ONE (Lan, Zhu, and Gong 2018) use the same kind of network for on-line training of peer networks with mutual KLdivergence losses.”
Large scale distributed neural network training through online distillation (2018, methodology): “Zhang et al. (2017) The weights learned by stochastic gradient descent in the non-convex setting will depend on the initialization, data presentation order, and the general vicissitudes of the infrastructure, especially when parallelization is involved.”
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning (2022, background): “At the same time, because of its similarity to human learning style, it is gradually applied to mutual learning [13], continual learning and other fields.”
Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, methodology): “We compare our method with knowledge distillation methods and deep mutual learning (Zhang et al., 2018), and also report the results of Online Distillation from Checkpoints (ODC) (Wei et al., 2019) for comparison.”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, methodology): “Following the analysis in Zhang et al. (2018), we add independent Gaussian noise of increasing strength to the parameters of the trained model and analyze the change in accuracy and loss across the training samples.”
CPR: Classifier-Projection Regularization for Continual Learning (2020, methodology): “…with wide local minima-promoting regularizers achieve better generalization and higher accuracy (Keskar et al., 2017; Pereyra et al., 2017; Chaudhari et al., 2019; Zhang et al., 2018) , and can be more robust to weight perturbations (Zhang et al., 2018) when compared to usual training methods.”
Knowledge Distillation: A Survey (2020, methodology): “ion is proposed to further improve the performance of the student model, Knowledge Distillation: A Survey 9 especially when the high-capacity teacher model with powerful performance is not available (Zhang et al., 2018b; Chen et al., 2019a). In online distillation, both the teacher model and the stu”
Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, background): “Firstly, as done in [43, 44], we consider the model at the end of training and add independent Gaussian noise with growing σ to each parameter.”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, methodology): “Our work differs from Zhang et al. (2017) and Anil et al. (2018) in that they collaboratively train multiple student models with codistillation, while we use multiple teacher models to train a single student model, the multilingual NMT model.”
In Defense of the Triplet Loss for Person Re-Identification (2017, methodology): “We do not include approaches which are orthogonal to ours and could be integrated in a straightforward manner, such as various re-ranking schemes, data augmentation, and regularization [3, 43, 54, 7, 47, 53, 56].”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “The teacher network is sometimes deeper and wider (Hinton, Vinyals, and Dean 2015), of similar size (Anil et al. 2018; Zhang et al. 2017), or shallower but wider (Romero et al.”
Knowledge Distillation by On-the-Fly Native Ensemble (2018, methodology): “There is no complexity of asynchronous updating among different networks which is required in deep mutual learning [16].”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, methodology): “As reported by several other authors [11, 38], we train the student network by simultaneously considering the standard metric learning loss LML (see Eq.”
On the Efficacy of Knowledge Distillation (2019, background): “But does knowledge distillation fulfill this promise of generality? Unfortunately, in spite of the recent interest in variants of knowledge distillation [17, 24, 23, 21, 27, 26, 5, 14], an empirical answer to this question is missing.”
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, methodology): “[43], we conduct the following experiments to show that the proposed self distillation framework can converge to a flat minimun.”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “We add a simple L2 loss to align these two value heads in a co-distillation manner (Zhang et al., 2018): align := ‖v𝑡 − v̂𝑡 ‖2.”