Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning

By Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren. 2017

262 citations recorded by the source corpus.

The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data samples. In this paper, we propose a novel method, called Smooth Neighbors on Teacher Graphs (SNTG). In SNTG, a graph is constructed based on the predictions of the teacher model, i.e., the implicit self-ensemble of models. Then the graph serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold. We achieve state-of-the-art results on semi-supervised learning benchmarks. The error rates are 9.89%, 3.99% for CIFAR-10 with 4000 labels, SVHN with 500 labels, respectively. In particular, the improvements are significant when the labels are fewer. For the non-augmented MNIST with only 20 labels, the error rate is reduced from previous 4.81% to 1.36%. Our method also shows robustness to noisy labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Graph Learning

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, neutral): “Works in this category include Mean Teacher [5], fast-Stochastic Weight Averaging [33] and Smooth Neighbors on Teacher Graphs [29].”
ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning (2021, methodology): “Because mean teacher (MT) [41] can be easily extended to many recent works [9, 2, 21], we choose it as the base model of SSL classifier in ORDisCo.”
Incremental Object Detection via Meta-Learning (2020, neutral): “This has found success in semi-supervised learning [50], [51], [52] and domain adaptation [53], [54], [55], [56] settings.”
Interpolation Consistency Training for Semi-Supervised Learning (2019, methodology): “Results We provide the results for CIFAR10 and SVHN datasets using CNN-13 architecture in Tables 1 and 2, respectively.”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, neutral): “, 2017)(2), (Laine and Aila, 2016)(3) and (Luo et al., 2018)(4).”
Milking CowMask for Semi-Supervised Image Classification (2020, background): “, 2020) analyzed semantic segmentation problems, finding that they exhibit a challenging data distribution where the cluster assumption – identified in prior work (Luo et al., 2018; Sajjadi et al., 2016a; Shu et al., 2018; Verma et al., 2019) as important to the success of consistency regularization”
Confidence Regularized Self-Training (2019, background): “Yet few principled research have considered regularized self-training within the context of SSL/UDA.”
Cluster Alignment With a Teacher for Unsupervised Domain Adaptation (2019, methodology): “Formally, resembling the effective SNTG loss in [25], we employ the following discriminative clustering loss (we omit the dependence of f on θ for clarity, unless stated otherwise):”
Certainty-Driven Consistency Loss for Semi-supervised Learning (2019, methodology): “Several recently proposed methods are based on training the model predictions to be consistent to perturbations [11, 12, 13, 14, 3, 15, 16].”
Towards Discriminability and Diversity: Batch Nuclear-Norm Maximization Under Label Insufficient Situations (2020, unknown): “51 SNTG+Π-model [28] - 62.”
Pairwise Teacher-Student Network for Semi-Supervised Hashing (2019, unknown): “However, the graph in SNTG is built speciﬁcally for classi-ﬁcation.”
In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning (2021, methodology): “For CIFAR-10 and CIFAR-100 experiments, we use the CNN-13 architecture that is commonly used to benchmark SSL methods (Oliver et al., 2018a; Luo et al., 2018).”
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning (2019, methodology): “work that makes use of the 13-CNN [18] in CIFAR-10/100: Π model [30], TE [30], MT [18], Π model-SN [22], MADNN [31], Deep-Co [21], TSSDL [19], LP [17], CCL [11], fast-SWA [29] and ICT [28].”
Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning (2020, methodology): “Different uncertainty metrics have been explored in the previous literature, including choosing the samples with the highest-conﬁdence (Zhu 2006), or retrieving the nearest samples in feature space (Shi et al. 2018; Iscen et al. 2019; Luo et al. 2017).”
Transductive Semi-Supervised Deep Learning Using Min-Max Features (2018, background): “Recent research works [24–28] have explored supervisory information from unlabeled image samples by adding diﬀerent perturbations to each image, and enforcing the label consistency between diﬀerent perturbed versions of the image.”