Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation

By George Papandreou, Liang-Chieh Chen, Kevin P. Murphy, Alan L. Yuille. 2015

972 citations recorded by the source corpus.

Deep convolutional neural networks (DCNNs) trained on a large number of images with strong pixel-level annotations have recently significantly pushed the state-of-art in semantic image segmentation. We study the more challenging problem of learning DCNNs for semantic image segmentation from either (1) weakly annotated training data such as bounding boxes or image-level labels or (2) a combination of few strongly labeled and many weakly labeled images, sourced from one or multiple datasets. We develop Expectation-Maximization (EM) methods for semantic image segmentation model training under these weakly supervised and semi-supervised settings. Extensive experimental evaluation shows that the proposed techniques can learn models delivering competitive results on the challenging PASCAL VOC 2012 image segmentation benchmark, while requiring significantly less annotation effort. We share source code implementing the proposed system at https://bitbucket.org/deeplab/deeplab-public.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks (2018, background): “…(RNN) neural networks have enjoyed great success in many real-world problems, including scene classification Krizhevsky et al., semantic segmentation Papandreou et al. (2015), speech recognition Hinton et al. (2012), gaming Mnih et al. (2013; 2015), and image captioning Johnson et al. (2016).”
DeepSetNet: Predicting Sets with Deep Neural Networks (2016, background): “Deep neural networks have shown state-of-the-art performance on many computer vision problems, including semantic segmentation [33], visual tracking [31], image captioning [21], scene classification [22], and object detection [27].”
Extreme Clicking for Efficient Object Annotation (2017, methodology): “We train DeepLab [10, 48] based on VGG-16 [58] on the VOC 2012 train set (5,623 images) and then we test on its val set (1,449 images).”
Pyramid Scene Parsing Network (2016, methodology): “Following the procedure of [26, 7, 31, 3], we use augmented data with the annotation of [10] resulting 10,582, 1,449 and 1,456 images for training, validation and testing.”
Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “To address this issue, Weakly Supervised Semantic Segmentation (WSSS) methods aim to obtain effective segmentation models using cheaper supervisions such as bounding boxes [18, 32, 49], scribbles [39, 60], points [6, 53], and image-level labels [31, 34, 35, 59].”
An EM Framework for Online Incremental Learning of Semantic Segmentation (2021, positive): “Re-labeling, also known as self-training [16, 25, 33], is a strategy to learn from partially labeled or unlabeled data.”
Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, methodology): “We can use weakly supervised annotation such as using bounding boxes to reduce manual annotation cost (Pathak, Krahenbuhl, and Darrell 2015; Papandreou et al. 2015), but even obtaining weakly annotated data for semantic segmentation can be time-consuming.”
The limits and potentials of deep learning for robotics (2018, background): “Semi-supervised approaches [56], [82], [89] that can leverage unlabeled data are of particular interest.”
Monocular Object Instance Segmentation and Depth Ordering with CNNs (2015, methodology): “We refer the reader to Fig.”
Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform (2015, methodology): “We perform another experiment with the stronger baseline of [34], where DeepLab is pretrained with the MS-COCO 2014 dataset [29].”
What's the Point: Semantic Segmentation with Point Supervision (2015, methodology): “The most common setting is where only image-level labels for the presence or absence of classes are provided during training [4,20,21,22,23,24,25], but other forms of weak super-vision have been explored as well, such as bounding box annotations [4], eye tracks [26], free-form squiggles [17,18], or…”
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes (2016, neutral): “Published Unpublished LRR[20] Adelaide[34] Dilation[55] ENet[44] SegNet[2] DeepLab[43] FRRNA/B Dilation[55] LRR[20] FRRNB Figure 6.”
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation (2016, methodology): “An improved method was suggested by [24] who use an EM algorithm to better model global properties of the image segments.”