Semi-Supervised Learning by Label Gradient Alignment

By Jacob Jackson, John Schulman. 2019

20 citations recorded by the source corpus.

We present label gradient alignment, a novel algorithm for semi-supervised learning which imputes labels for the unlabeled data and trains on the imputed labels. We define a semantically meaningful distance metric on the input space by mapping a point (x, y) to the gradient of the model at (x, y). We then formulate an optimization problem whose objective is to minimize the distance between the labeled and the unlabeled data in this space, and we solve it by gradient descent on the imputed labels. We evaluate label gradient alignment using the standardized architecture introduced by Oliver et al. (2018) and demonstrate state-of-the-art accuracy in semi-supervised CIFAR-10 classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “In Appendix C, we also compare UDA with recently proposed methods including ICT [61] and mixmixup [19] which enforce interpolation smootheness similar to mixup [69] and LGA + VAT [28], an algorithm based on gradient similarity.”
Meta Pseudo Labels (2020, background): “On the other hand, label propagation losses typically enforce that the model is invariant against certain transformations of the data such as data augmentations, adversarial attacks, or proximity in the latent space [35, 64, 44, 5, 76, 30, 71, 58, 32, 51, 20].”
Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning (2020, background): “2017), moderate-to-high data augmentation (Miyato et al. 2018; Jackson and Schulman 2019; Verma et al. 2019; Berthelot et al. 2019), and heavy data augmentation (Xie et al.”
Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples (2021, unknown): “…Methods using label propagation: Temporal Ensemble [47] 300 83.6 ± 0.6 Mean Teacher [8] 300 84.1 ± 0.3 VAT + EntMin [13] 123 86.9 ± 0.4 LGA + VAT [48] – 87.9 ± 0.2 ICT [14] 600 92.7 ± 0.0 MixMatch [9] – 93.8 ± 0.1 ReMixMatch [10] – 94.9 ± 0.0 EnAET [49] 1024 94.7 ± UDA [15, 6] 2564 94.5 ± 0.2…”
Unsupervised Data Augmentation (2019, methodology): “…and Valpola, 2017], which enforces model parameter smoothness, ICT [Verma et al., 2019] and mixmixup [Hataya and Nakayama, 2019] which enforce interpolation smootheness similar to mixup [Zhang et al., 2017] and LGA + VAT [Jackson and Schulman, 2019], an algorithm based on gradient similarity.”