Label Propagation for Deep Semi-Supervised Learning

By Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Ondřej Chum. 2019

731 citations recorded by the source corpus.

Semi-supervised learning is becoming increasingly important because it can combine data carefully labeled by humans with abundant unlabeled data to train deep neural networks. Classic methods on semi-supervised learning that have focused on transductive learning have not been fully exploited in the inductive framework followed by modern deep learning. The same holds for the manifold assumption---that similar examples should get the same prediction. In this work, we employ a transductive label propagation method that is based on the manifold assumption to make predictions on the entire dataset and use these predictions to generate pseudo-labels for the unlabeled data and train a deep neural network. At the core of the transductive method lies a nearest neighbor graph of the dataset that we create based on the embeddings of the same network. Therefore our learning process iterates between these two steps. We improve performance on several datasets especially in the few labels regime and show that our work is complementary to current state of the art.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “Works based on pseudo label [37, 31, 60, 1] are similar to self-training, but also suffers the same problem with consistency training, since it relies on a model being trained instead of a converged model with high accuracy to generate pseudo labels.”
ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning (2021, methodology): “Because mean teacher (MT) [41] can be easily extended to many recent works [9, 2, 21], we choose it as the base model of SSL classifier in ORDisCo.”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, methodology): “These methods can be semi-supervised [24,30,49], self-supervised [5,17,39] or perform few-shot learning [12,13,48].”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, background): “[2], other semi-supervised self-training in semantic segmentation [30, 31, 32, 33, 34], and other approaches based on pseudo label such as [4, 35, 36, 3].”
Extract Free Dense Labels from CLIP (2021, methodology): “Self-training has firstly become popular in the semisupervised classification task [33,29,35,43] and is also recently applied in the semi-supervised/zero-shot semantic segmentation settings [27,37,4,28,40,36,56,8].”
Triple Generative Adversarial Networks (2019, methodology): “Among them,mean teacher [32] is a competitive method and many recent methods [40], [41], [44], [45] are built upon it.”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, methodology): “In particular, our model is trained with some human-annotated images and abundant pseudolabeled [42,72,34,3] video sequences.”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, neutral): “Self-training has ﬁrstly become popular in the semi-supervised classiﬁcation task [29, 33, 35, 43] and is also recently applied in the semi-supervised/zero-shot semantic segmentation settings [4, 10, 27, 28, 36, 37, 40, 56].”
SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations (2020, methodology): “2019) which are based on data augmentation, Π-model (Laine and Aila 2017) and Mean Teacher (Tarvainen and Valpola 2017), based on model consistency regularization, Label Propagation (LP) (Iscen et al. 2019) based on pseudolabel and CT-GAN (Wei et al.”
Decoupled Certainty-Driven Consistency Loss for Semi-supervised Learning (2019, methodology): “Deep label propagation [1] combines mean teacher with label diffusion to get further improvement.”
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval (2021, background): “Existing semi-supervised classification approach usually considers the probability distribution over classes to filter out noisy samples based on a predefined threshold [44], top-K selection [36], or uses entropy-based instance-wise weighting [18] to deal with noisy synthetic labels.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, methodology): “Graph-based prediction regularization operates in the prediction space [63], [108], as in Label Propagation [63].”
Conditional Negative Sampling for Contrastive Learning of Visual Representations (2020, background): “Several later approaches (Zhuang et al., 2019; Tian et al., 2019; Iscen et al., 2019; Srinivas et al., 2020) build on the IR framework.”
An Overview of Deep Semi-Supervised Learning (2020, background): “Pseudo-labeling [91, 2, 72, 139], similar to self-training, the objective of pseudo-labeling is to generate proxy labels to enhance the learning process.”
Generating Accurate Pseudo-Labels in Semi-Supervised Learning and Avoiding Overconfident Predictions via Hermite Polynomial Activations (2019, background): “These Pseudo-label (PL) based SSL methods serve as a way to execute transductive learning [13, 24], where label inference on the given test dataset is more important for the user as compared to using the trained model later, on other unseen samples from other data sources.”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, methodology): “Semi-supervised learning aims to utilize unlabeled data to boost model performance, and is studied in computer vision (Iscen et al., 2019), item ranking (Park and Chang, 2019; Huang et al., 2019b), and multi-label classiﬁcation (Miyato et al., 2015; Wang et al., 2018, 2019b).”
A Unified Objective for Novel Class Discovery (2021, background): “Clustering objectives are generally based on pseudolabels [7, 12, 14, 29, 30, 31] estimated on the unlabeled set.”