Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency

By Sudhanshu Mittal, Maxim Tatarchenko, Thomas Brox. 2019

469 citations recorded by the source corpus.

The ability to understand visual information from limited labeled data is an important aspect of machine learning. While image-level classification has been extensively studied in a semi-supervised setting, dense pixel-level classification with limited data has only drawn attention recently. In this work, we propose an approach for semi-supervised semantic segmentation that learns from limited pixel-wise annotated samples while exploiting additional annotation-free images. The proposed approach relies on adversarial training with a feature matching loss to learn from unlabeled images. It uses two network branches that link semi-supervised classification with semi-supervised segmentation including self-training. The dual-branch approach reduces both the low-level and the high-level artifacts typical when training with few labels. The approach attains significant improvement over existing methods, especially when trained with very few labeled samples. On several standard benchmarks—PASCAL VOC 2012, PASCAL-Context, and Cityscapes—the approach achieves new state-of-the-art in semi-supervised learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, background): “Semi-supervised segmentation methods have been proposed to address such limitations by jointly learning from both annotated and unannotated images [11, 22, 29, 40, 42, 45].”
DACS: Domain Adaptation via Cross-domain Mixed Sampling (2020, background): “ annotate some data from the domain of interest and re-train (ﬁne-tune) the network on this new data. Additionally, Semi-Supervised Learning (SSL) methods for semantic segmentation have been proposed [4, 5, 6, 7, 8, 9], effectively training on a small amount of labelled data by relying on complement”
ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation (2021, methodology): “Motivated by the tremendous progress in SSL, recent works on semi-supervised semantic segmentation are evolving from GANs-based methods [17, 44, 21, 32] to integrating the advanced SSL methods into the segmentation scenario, such as enforcing consistent predictions of the same unlabeled image but un”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, methodology): “While SSL approaches have been more widely explored for classification, recent methods also tackle pixel-wise tasks [37, 62, 40, 47, 22, 68].”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, unknown): “Self-training has ﬁrstly become popular in the semi-supervised classiﬁcation task [29, 33, 35, 43] and is also recently applied in the semi-supervised/zero-shot semantic segmentation settings [4, 10, 27, 28, 36, 37, 40, 56].”
Bootstrapping Semantic Segmentation with Regional Contrast (2021, methodology): “In this work, we standardise the baselines and implement four strong semi-supervised segmentation methods ourselves: S4GAN (Mittal et al., 2019): an adversarial learning based semi-supervised method; CutOut (French et al., 2020), CutMix (French et al., 2020), ClassMix (Olsson et al., 2021): three im”
Extract Free Dense Labels from CLIP (2021, methodology): “Self-training has firstly become popular in the semisupervised classification task [33,29,35,43] and is also recently applied in the semi-supervised/zero-shot semantic segmentation settings [27,37,4,28,40,36,56,8].”
SemiCDNet: A Semisupervised Convolutional Neural Network for Change Detection in High Resolution Remote-Sensing Images (2020, methodology): “In [57], a semisupervised semantic segmentation method is proposed by using two network branches, which encourage high-and low-level consistency when training with few labeled images.”
Semi-supervised Segmentation Based on Error-Correcting Supervision (2020, methodology): “The work of Mittal et al. [23] extends [22] for semi-supervised learning.”
Semi-supervised semantic segmentation needs strong, varied perturbations (2019, methodology): “[24] employ GAN-based adversarial learning, using a discriminator network that distinguishes real from predicted segmentation maps to guide learning.”
Uncertainty-aware Joint Salient Object and Camouflaged Object Detection (2021, background): “[18, 36] treated adversarial learning as a confidence measure to obtain the trustworthy regions of semantic segmentation predictions of unlabeled data for semi-supervised learning.”
Mutual consistency learning for semi-supervised medical image segmentation (2021, methodology): “training (Miyato et al., 2018; Mittal et al., 2019; You et al., 2022b) is used to enforce stronger consistency constraints for the model training.”
PseudoSeg: Designing Pseudo Labels for Semantic Segmentation (2020, methodology): “Early methods utilize a GAN-based model either to generate additional training data (Souly et al., 2017) or to learn a discriminator between the prediction and the ground truth mask (Hung et al., 2018; Mittal et al., 2019).”
A Survey on Deep Semi-Supervised Learning (2021, background): “Semantic segmentation PASCAL VOC [65], PASCAL context [80], MS COCO [66], Cityscapes [81], CamVid [82], SiftFlow [83], [84], StanfordBG [85] [50], [86], [87], [88], [89], [90], [91], [92], [93], [94]”
Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation (2022, methodology): “Earlier works [47, 56] incorporate the GANs [21] as an auxiliary supervision for unlabeled images via discriminating pseudo labels from manual labels.”
Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision (2021, methodology): “Recently it is applied for semi-supervised segmentation [6, 43, 42, 9, 13, 25, 14, 24].”
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort (2021, methodology): “Semi-Supervised Learning: Given a large set of unlabeled images and a small set of annotated images, semisupervised approaches [37, 47, 22, 40, 27] aim to learn better segmentation networks than with supervised data alone.”
Guided Collaborative Training for Pixel-wise Semi-Supervised Learning (2020, methodology): “[33] modify Mean Teacher [44] to a multi-label classifier and use it as a filter to remove uncertain categories.”