Improving Consistency-Based Semi-Supervised Learning with Weight Averaging

By Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, Andrew Gordon Wilson. 2018

36 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “CIFAR-10 and CIFAR-100 with a larger model Some prior work [44, 2] has also considered the use of a larger, 26 million-parameter model.”
Averaging Weights Leads to Wider Optima and Better Generalization (2018, background): “I Two papers at UDL workshop tomorrow! I Improving Stability in Deep Reinforcement Learning with Weight Averaging I Fast Uncertainty Estimates and Bayesian Model Averaging of DNNs I Athiwaratkun et al. [2018]: use a modified version of SWA to get SOTA results in Semi-Supervised Learning”
The Unusual Effectiveness of Averaging in GAN Training (2018, methodology): “Averaging at maximum likelihood objective: Uniform Average and Exponential Moving Average over model parameters has been used for a long time (Polyak & Juditsky, 1992; van den Oord et al., 2017; Athiwaratkun et al., 2018; Izmailov et al., 2018).”
Improving Stability in Deep Reinforcement Learning with Weight Averaging (2018, methodology): “SWA is a training technique based on averaging weights of the models collected during training, which was shown to improve generalization for both supervised and semi-supervised [Athiwaratkun et al., 2018] learning.”
Exploring Object Relation in Mean Teacher for Cross-Domain Detection (2019, methodology): “Inspired by the recent success of consistency-based methods in semi-supervised learning [1, 20, 48] and Mean Teacher in cross-domain recognition [9], we formulate our cross-domain detection model in a Mean Teacher paradigm by enforcing the predictions of teacher and student models consistent under p”
Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency (2019, methodology): “We find that consistency-based semi-supervised classification methods [2], [37] show improvement over the supervised baseline while satisfying at least two procedures mentioned by Oliver et al.”
Fast Uncertainty Estimates and Bayesian Model Averaging of DNNs (2018, background): “In the experiments performed in [5] and [2], SWA outperforms other training techniques at minimal computation cost, producing state-of-the-art results on semi-supervised learning tasks.”