Weakly Supervised Semantic Segmentation Using Web-Crawled Videos

By Seunghoon Hong, Donghun Yeo, Suha Kwak, Honglak Lee, Bohyung Han. 2017

161 citations recorded by the source corpus.

We propose a novel algorithm for weakly supervised semantic segmentation based on image-level class labels only. In weakly supervised setting, it is commonly observed that trained model overly focuses on discriminative parts rather than the entire object area. Our goal is to overcome this limitation with no additional human intervention by retrieving videos relevant to target class labels from web repository, and generating segmentation labels from the retrieved videos to simulate strong supervision for semantic segmentation. During this process, we take advantage of image classification with discriminative localization technique to reject false alarms in retrieved videos and identify relevant spatio-temporal volumes within retrieved videos. Although the entire procedure does not require any additional supervision, the segmentation annotations obtained from videos are sufficiently strong to learn a model for semantic segmentation. The proposed algorithm substantially outperforms existing methods based on the same level of supervision and is even as competitive as the approaches relying on extra annotations.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

RECALL: Replay-based Continual Learning in Semantic Segmentation (2021, methodology): “Recently, it was employed also in semantic segmentation to provide a plentiful source of both images [20, 38] and videos [19] with weak image-level class labels during training.”
Learning to Play with Intrinsically-Motivated Self-Aware Agents (2018, neutral): “More explicit self-supervised auxiliary tasks include semantic segmentation (Hong et al., 2017), pose estimation (Mitash et al.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “Learning from weak supervision: Given the known challenges in gathering exhaustive annotations for various image and video tasks, leveraging object labels and other meta information to supply weak supervision [7, 15, 19, 22, 30, 31, 39, 40, 50, 52, 53, 54, 55, 56, 59] is a widely-adopted approach.”
PreDet: Large-scale weakly supervised pre-training for detection (2021, background): “Learning detection and semantic segmentation models [9, 8, 53, 57, 21, 46, 47] directly from web-scale data in a weakly-supervised manner has also shown some promise.”
Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation (2020, methodology): “[18] CVPR17 55.”
Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation (2021, methodology): “Compared to bounding boxes [14, 26, 43], points [4], and scribbles [33, 46], image-level labels [29, 51, 21, 8, 23, 2, 52, 24] are the most widely used weak annotations due to their easy availability.”
Single-Stage Semantic Segmentation From Image Labels (2020, methodology): “We significantly improve over previous single-stage methods [38, 41, 43], as well as outperform the single-stage WebCrawl [18], which relies on additional training data and needs multiple forward passes through its class-agnostic decoder.”
Weakly Supervised Segmentation with Maximum Bipartite Graph Matching (2020, methodology): “Other methods adopting motion videos [16], instance saliency mask [11] also yield promising results.”
Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi-Supervised Semantic Segmentation (2018, background): “[8] achieved the state-of-the-art performance on this challenging task.”
Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations (2019, background): “In particular, image-level class labels have been widely used as weak labels since they require minimal or no effort for annotation [1, 12, 21, 22, 38, 42, 43, 48, 53].”
Adversarial Learning for Semi-supervised Semantic Segmentation (2018, background): “In addition to image-level supervisions, the segmentation network can also be trained with bounding boxes [6, 19], point supervision [2], or web videos [16].”
Identity-Guided Human Semantic Parsing for Person Re-Identification (2020, background): “semantic segmentation [16, 47, 19, 12, 11, 33, 44, 62], which aim to locate objects like person, horse or dog at pixel-level with image-level supervision.”
Small Sample Learning in Big Data Era (2018, methodology): “For example, Hong et al. (2017b) made use of web videos as additional data, while the annotations of web videos returned by a search engine tend to be inevitably noisy since the query keywords may not be consistent with the visual content of target images, and thus the problem is evidently weakly…”
Self-Erasing Network for Integral Object Attention (2018, background): “As only image-level labels are available, most recent approaches [16, 3, 12, 34, 9], more or less, rely on different attention models due to their ability of covering small but discriminative semantic regions.”
Bootstrapping the Performance of Webly Supervised Semantic Segmentation (2018, methodology): “Different from existing filtering approaches [30, 10], our method is based on dense masks and provides richer information of the images.”
FickleNet: Weakly and Semi-Supervised Semantic Image Segmentation Using Stochastic Inference (2019, methodology): “These methods include TransferNet [9], which was trained on pixel-level annotations of 60 classes (not Pascal VOC classes) of COCO [20] images, and CrawlSeg [10], which Table 3.”
Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation (2018, methodology): “Figure 6 presents qualitative results of Ours-ResNet38 and compares them to those of CrawlSeg [11], which is the current state of the art using image-level supervision.”
Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data (2019, background): “[12] propose an approach for semantic segmentation, starting with a set of weakly annotated images and extending the system in a second set to learn from web video data.”