STC: A Simple to Complex Framework for Weakly-Supervised Semantic Segmentation

By Yunchao Wei, Xiaodan Liang, Yunpeng Chen, Xiaohui Shen, Ming-Ming Cheng, Yao Zhao, Shuicheng Yan. 2015

585 citations recorded by the source corpus.

Recently, significant improvement has been made on semantic object segmentation due to the development of deep convolutional neural networks (DCNNs). Training such a DCNN usually relies on a large number of images with pixel-level segmentation masks, and annotating these images is very costly in terms of both finance and human effort. In this paper, we propose a simple to complex (STC) framework in which only image-level annotations are utilized to learn DCNNs for semantic segmentation. Specifically, we first train an initial segmentation network called Initial-DCNN with the saliency maps of simple images (i.e., those with a single category of major object(s) and clean background). These saliency maps can be automatically obtained by existing bottom-up salient object detection techniques, where no supervision information is needed. Then, a better network called Enhanced-DCNN is learned with supervision from the predicted segmentation masks of simple images based on the Initial-DCNN as well as the image-level annotations. Finally, more pixel-level segmentation masks of complex images (two or more categories of objects with cluttered background), which are inferred by using Enhanced-DCNN and image-level annotations, are utilized as the supervision information to learn the Powerful-DCNN for semantic segmentation. Our method utilizes 40K simple images from Flickr.com and 10K complex images from PASCAL VOC for step-wisely boosting the segmentation network. Extensive experimental results on PASCAL VOC 2012 segmentation benchmark well demonstrate the superiority of the proposed STC framework compared with other state-of-the-arts.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “Note that saliency priors, used by many works such as [10, 31, 32] on Pascal VOC, are not suitable for “stuff” classes as popular saliency datasets [34–36] only consider “things” to be salient.”
Contrast Prior and Fluid Pyramid Integration for RGBD Salient Object Detection (2019, background): “…including video/image segmentation [17, 40], object recognition [46], visual tracking [ 3], foreground maps evaluation [14, 15], image retrieval [6,16,22,38], content-aware image editing [8], information discovery [58], photo synthesis [5, 29], and weakly supervised semantic segmentation [52].”
Deeply Supervised Salient Object Detection with Short Connections (2016, background): “Such a useful property allows salient object detection to commonly serve as the first step to a variety of computer vision applications including image and video compression [2], [3], image segmentation [4], content-aware image editing [5], [6], object recognition [7], weakly supervsied segmantic se”
Tell Me Where to Look: Guided Attention Inference Network (2018, methodology): “For methods purely using image-level labels, we compare our GAIN-based SEC (denoted as GAIN in the table) with SEC [12], AE-PSL [31], TPL [11], STC [32] and etc.”
Effective Use of Synthetic Data for Urban Scene Semantic Segmentation (2018, background): “Weakly-supervised semantic segmentation aims to leverage a weaker form of annotation, such as image tags [23, 30, 32, 34, 40, 41, 49, 48, 29], bounding boxes [8, 21], scribbles [2] and object size statistics [31], which are cheaper to obtain.”
Discovering Class-Specific Pixels for Weakly-Supervised Semantic Segmentation (2017, methodology): “[33] proposed a simple to complex framework where a network is first trained using simple images (single object category) followed by training over complex ones (multiple objects).”
Regional Interactive Image Segmentation Networks (2017, unknown): “tion [18, 5, 30, 27, 28].”
Self-produced Guidance for Weakly-supervised Object Localization (2018, methodology): “Weakly Supervised Learning (WSL) has been successfully applied on many tasks, such as object localization [1,2,3,4,5,6,7], relation detection [8] and semantic segmentation [9,10,11,12,13].”
Interactive Object Segmentation With Inside-Outside Guidance (2020, methodology): “Particularly, image-level labels [50, 61, 63, 62, 27, 32], points [3, 51], bounding boxes [18, 33] and scribbles [39, 58, 59] have been employed as guidance to supervise the training of semantic segmentation networks.”
Simple Does It: Weakly Supervised Instance and Semantic Segmentation (2016, background): “In order to keep annotation cost low, recent work has explored different forms of supervision for semantic labelling: image labels [29, 28, 27, 30, 42], points [3], scribbles [44, 23], and bounding boxes [8, 27].”
Saliency Guided Self-Attention Network for Weakly and Semi-Supervised Semantic Segmentation (2019, background): “…images containing TABLE 7: Inﬂuence of saliency detectors to the segmentation performance on PASCAL VOC 2012 validation set. saliency mask S-Net [29] A typical failure case is also presented in the last row, in which the dining table is indistinguishable from the background and thus…”
Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground (2018, methodology): “ions in many other computer vision tasks including: visual tracking [4], image retrieval [14,16], computer graphics [9], content aware image resizing [45], and weakly supervised semantic segmentation [18,39,40]. ? M.M. Cheng (cmm@nankai.edu.cn) is the corresponding author. This research was supporte”
Weakly Supervised Instance Segmentation Using Class Peak Response (2018, background): “Although effective, these approaches require signiﬁcant more human efforts than image-level supervised methods [24, 25, 41, 14, 33].”
Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation (2021, methodology): “One promising approach to address the annotation problem for semantic segmentation is to learn from weak labels, such as image-level annotations [29, 51, 21, 8, 23, 2, 52, 24], bounding boxes [14, 26, 43], points [4], and scribbles [33, 46].”
Structure-Measure: A New Way to Evaluate Foreground Maps (2017, unknown): “The evaluation of a predicted foreground map against a ground-truth (GT) annotation map is crucial in evaluating and comparing various computer vision algorithm for applications such as object detection [6, 8, 24, 40], saliency prediction [5, 20, 42], image segmentation [41], contentbased image retr”
Interactive Object Segmentation With Inside-Outside Guidance (2022, methodology): “Particularly, image-level labels [67], [68], [69], [70], [71], [72], points [73], [74], bounding boxes [75], [76], scribbles [77], [78], [79] have been employed as guidance to”