Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

By Antti Tarvainen, Harri Valpola. 2017

5,726 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Optimization & Training

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, background): “For example, Mean Teacher [58] maintains a teacher model with parameters being the ensemble of a student model’s parameters and enforces the consistency between the predictions by the teacher model and the student model.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “Apart from self-training, another important line of work in semi-supervised learning [9, 85] is based on consistency training [6, 4, 53, 36, 70, 45, 41, 51, 10, 12, 49, 2, 38, 72, 74, 5, 81].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, neutral): “4 Mean teacher [39] ResNeXt-152 - - - 90.”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “However, in our framework, its role differs since we do not have a queue nor a contrastive loss, and may be closer to the role of the mean teacher used in self-training [52].”
Flow Contrastive Estimation of Energy-Based Models (2019, neutral): “32) Mean Teacher [68] 5.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “Unlike past work using MixUp for SSL [43, 44, 18], we “mix” labeled examples with unlabeled examples and vice versa which we ﬁnd results in improved performance (section 4.2.3).”
Reward learning from human preferences and demonstrations in Atari (2018, methodology): “We tried applying the state of the art technique by Tarvainen and Valpola (2017) without much improvement in sample complexity while facing more training stability issues.”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “Similar ideas have been explored in semi-supervised learning [35], label noise learning [36], and very recently in contrastive learning [37].”
AutoAugment: Learning Augmentation Policies from Data (2018, methodology): “Similar to the experimental convention used by the semi-supervised learning community [62–66] we train on 4,000 labeled samples. but we do not use the 46,000 unlabeled samples during training.”
Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (2022, methodology): “An efﬁcient method for aggregating the weights of a model is provided by Mean Teacher (Tarvainen & Valpola, 2017) which is a knowledge distillation approach that uses an exponential moving average (EMA) of the student’s weights during training as a teacher for semi-supervised learning.”
Knowledge Distillation: A Survey (2020, background): “…operations (Howard et al. 2017; Zhang et al. 2018a; Huang et al. 2017); or (4) a small network with optimized global network structure (Liu et al. 2019i; Xie et al. 2020; Gu and Tresp 2020); or (5) the same network as teacher (Zhang et al. 2018b; Furlanello et al. 2018; Tarvainen and Valpola 2017).”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, background): “ that they generally require large amounts of labeled data for training. Some recent methods [9,37] train neural networks with a small number of labeled examples. These methods can be semi-supervised [24,30,49], self-supervised [5,17,39] or perform few-shot learning [12,13,48]. However, they assume ”
Incremental Object Detection via Meta-Learning (2020, positive): “Using a Moving Average Teacher [50]: Mean-Teacher [50] proposes to use a moving average teacher to distill the student from.”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, methodology): “Mean Teacher Model The Mean Teacher model (MT) proposed in Tarvainen and Valpola (2017) uses the same data and weight perturbations as the Π model; however, the teacher weights wg are the exponential moving average (EMA) of the student weights wf : wkg = α · wk−1g + (1− α) · wkf .”
Relational Knowledge Distillation (2019, methodology): “Instead of imitating output activations of the teacher, Zagoruyko and Komodakis [47] and Huang and Wang [12] transfer an attention map of a teacher network into a student, and Tarvainen and Valpola [36] introduce a similar approach using mean weights.”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “Czarnecki et al. (2017) minimized the difference between teacher and student derivatives of the loss combined with the divergence from teacher predictions while Tarvainen and Valpola (2017) uses averaging model weights instead of target predictions.”
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring (2019, methodology): “Consistency regularization underlies most SSL methods (Miyato et al., 2018; Tarvainen & Valpola, 2017; Berthelot et al., 2019; Xie et al., 2019).”