Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation

By Alexander Kolesnikov, Christoph H. Lampert. 2016

818 citations recorded by the source corpus.

We introduce a new loss function for the weakly-supervised training of semantic image segmentation models based on three guiding principles: to seed with weak localization cues, to expand objects based on the information about which classes can occur in an image, and to constrain the segmentations to coincide with object boundaries. We show experimentally that training a deep convolutional neural network using the proposed loss function leads to substantially better segmentations than previous state-of-the-art methods on the challenging PASCAL VOC 2012 dataset. We furthermore give insight into the working mechanism of our method by a detailed experimental study that illustrates how the segmentation quality is affected by each term of the proposed loss function as well as their combinations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, methodology): “We use Grad-CAM localization as weak-supervision to train the segmentation architecture from SEC [26].”
Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “Following the standard methodology [3, 34], we augment the Pascal VOC dataset with images from [26] for a total of 10582 images for training and 1449 for validation annotated on 20 object categories.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “Learning from weak supervision: Given the known challenges in gathering exhaustive annotations for various image and video tasks, leveraging object labels and other meta information to supply weak supervision [7, 15, 19, 22, 30, 31, 39, 40, 50, 52, 53, 54, 55, 56, 59] is a widely-adopted approach.”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “Following recent work [42, 9, 13, 41], we use our weak supervision and image priors to approximate the ground-truth for a subset Ω ′ ⊂ Ω of the pixels in the image.”
Attention-Based Dropout Layer for Weakly Supervised Object Localization (2019, methodology): “Existing approaches mine and track discriminative features of each class for object detection [45, 36, 37, 9, 45, 25, 21, 41, 19, 2, 39, 15, 63, 7, 5, 4, 48, 14, 65, 32, 31, 58, 62, 8, 6] and segmentation [33, 29, 18, 16, 24, 52, 50].”
C-MIDN: Coupled Multiple Instance Detection Network With Segmentation Guidance for Weakly Supervised Object Detection (2019, background): “Kolesnikov and Lam-pert [16] introduce three principles and propose an end-to-end network to implement these principles.”
Webly Supervised Semantic Segmentation (2017, methodology): “They are usually much easier and faster to obtain than the other human annotations described above, and have thus been used in many weakly supervised semantic segmentation methods [13] [14] [15] [16] [17] [18] [19] [20] [21] [22].”
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation (2018, methodology): “[14] proposed an SEC approach that integrates three loss functions, i.”
Multi-class Token Transformer for Weakly Supervised Semantic Segmentation (2022, background): “To tackle this problem, speciﬁc segmentation losses have been proposed to cater for deﬁcient segmentation supervision, including SEC loss [19], CRF loss [35, 47] and contrastive loss [17].”
Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features (2018, methodology): “We compare our method with previous state-of-theart image-level weakly-supervised semantic segmentation methods: CCNN [20], EM-Adapt [19], MIL-sppxl [22], STC [31], DCSM [26], BFBP [25], AF-SS [23], SEC [13], CBTS [24] and AE-PSL [30].”
Self-Erasing Network for Integral Object Attention (2018, methodology): “Therefore, semantic segmentation with image-level supervision [25, 16, 35, 12, 34] is becoming a promising way to relief lots of human labors.”
Pseudo-mask Matters in Weakly-supervised Semantic Segmentation (2021, methodology): “To reduce the annotation burden, weaklysupervised semantic segmentation approaches have been proposed using scribble annotations [23, 32], bounding boxes [36, 9, 16] , points [3] or image-level labels [17, 27, 28, 2, 33].”
A Mutual Bootstrapping Model for Automated Skin Lesion Segmentation and Classification (2019, background): “Kolesnikov and Lampert [33] proposed the uniﬁed “seed, expand and con-strain” (SEC) framework, under which object localization cues, expansion and reﬁning boundary are integrated for segmentation.”
Effective Use of Synthetic Data for Urban Scene Semantic Segmentation (2018, background): “Weakly-supervised semantic segmentation aims to leverage a weaker form of annotation, such as image tags [23, 30, 32, 34, 40, 41, 49, 48, 29], bounding boxes [8, 21], scribbles [2] and object size statistics [31], which are cheaper to obtain.”