Semi-Supervised Semantic Segmentation With Cross-Consistency Training

By Yassine Ouali, Céline Hudelot, Myriam Tami. 2020

964 citations recorded by the source corpus.

In this paper, we present a novel cross-consistency based semi-supervised approach for semantic segmentation. Consistency training has proven to be a powerful semi-supervised learning framework for leveraging unlabeled data under the cluster assumption, in which the decision boundary should lie in low-density regions. In this work, we first observe that for semantic segmentation, the low-density regions are more apparent within the hidden representations than within the inputs. We thus propose cross-consistency training, where an invariance of the predictions is enforced over different perturbations applied to the outputs of the encoder. Concretely, a shared encoder and a main decoder are trained in a supervised manner using the available labeled examples. To leverage the unlabeled examples, we enforce a consistency between the main decoder predictions and those of the auxiliary decoders, taking as inputs different perturbed versions of the encoder's output, and consequently, improving the encoder's representations. The proposed method is simple and can easily be extended to use additional training signal, such as image-level labels or pixel-level labels across different domains. We perform an ablation study to tease apart the effectiveness of each component, and conduct extensive experiments to demonstrate that our method achieves state-of-the-art results in several datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, methodology): “Semi-supervised Experiments For a semi-supervised segmentation baseline, we used the ofﬁcial implementation of Regional Contrast [36] (ReCo) and CCT [45] meth-ods.”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, neutral): “Self-training has ﬁrstly become popular in the semi-supervised classiﬁcation task [29, 33, 35, 43] and is also recently applied in the semi-supervised/zero-shot semantic segmentation settings [4, 10, 27, 28, 36, 37, 40, 56].”
ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation (2021, methodology): “although our implementation of online pseudo labeling already surpasses some prior works [34, 31], its performance is still much inferior to our offline counterpart.”
PseudoSeg: Designing Pseudo Labels for Semantic Segmentation (2020, background): “Although we observe that many related works explore the second condition (i.e., augmentation) for image segmentation to enable consistency training framework (French et al., 2020; Ouali et al., 2020), we show that a wise design of pseudo labels for segmentation has great veiled potentials.”
A Survey on Deep Semi-Supervised Learning (2021, background): “Semantic segmentation PASCAL VOC [65], PASCAL context [80], MS COCO [66], Cityscapes [81], CamVid [82], SiftFlow [83], [84], StanfordBG [85] [50], [86], [87], [88], [89], [90], [91], [92], [93], [94]”
Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer (2021, methodology): “Then, we compared with eight previous semi-supervised methods: 1) Mean Teacher (MT) (Tarvainen and Valpola, 2017), 2) Entropy Minimization (EM) (Vu et al., 2019), 3) Deep Adversarial Network (DAN) (Zhang et al., 2017), 4) Uncertainty Aware Mean Teacher (UAMT) (Yu et al., 2019), 5) Interpolation Cons”
Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation (2022, methodology): “We follow [49] to set the hyper-parameters in these strategies.”
Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels (2022, methodology): “To alleviate this problem, many attempts [1,4,11,17,23,34,44,49] have been made towards semi-supervised semantic segmentation, which learns a model with only a few labeled samples and numerous unlabeled ones.”
ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning (2020, methodology): “Ouali et al. [7] propose to apply perturbations to the encoder’s output, where the cluster assumption is shown to hold.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, methodology): “More recently, SSL has been used for learning generic visual representations to facilitate many computer vision tasks such as image classiﬁcation [25], [26], image retrieval [27], object detection [28], [29], semantic segmentation [30], [31], [32], and pose estimation [33], [34], [35].”
Extract Free Dense Labels from CLIP (2021, methodology): “Self-training has firstly become popular in the semisupervised classification task [33,29,35,43] and is also recently applied in the semi-supervised/zero-shot semantic segmentation settings [27,37,4,28,40,36,56,8].”
Semi-supervised Left Atrium Segmentation with Mutual Consistency Training (2021, background): “Such consistency and entropy regularization facilitate the model to pay more attention to the unlabeled challenging and uncertain areas [9,11].”
Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision (2021, methodology): “, input perturbation by augmenting input images [11, 19], feature perturbation [27], and network perturbation [18].”
Bootstrapping Semantic Segmentation with Regional Contrast (2021, background): “Some solutions improved the quality of pseudo-labelling, using adversarial learning [12, 21] and class activation maps [38]; or enforcing consistency from different augmented images [9, 24], perturbed features [25] and different networks [13].”
Semi-supervised Medical Image Segmentation through Dual-task Consistency (2020, background): “Dual-task Consistency: In general semi-supervised learning, consistency losses are designed to encourage smooth predictions in a data-level, i.e. the predictions of same data under different transformations (Li et al. 2020) and perturbations (Ouali, Hudelot, and Tami 2020) should be the same.”
Mutual consistency learning for semi-supervised medical image segmentation (2021, neutral): “For example, Ouali et al. (2020) enforced several data augmentation operations to perturb the intermediate feature maps and constrained the model to output invariant segmentation maps.”
Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation (2022, positive): “These feature level perturbations are feature noise, feature dropout, object masking, context masking, guided cutout, intermediate VAT, and random dropout, based on the work in [57].”