Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach

By Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, Shuicheng Yan. 2017

871 citations recorded by the source corpus.

We investigate a principle way to progressively mine discriminative object regions using classification networks to address the weakly-supervised semantic segmentation problems. Classification networks are only responsive to small and sparse discriminative regions from the object of interest, which deviates from the requirement of the segmentation task that needs to localize dense, interior and integral regions for pixel-wise inference. To mitigate this gap, we propose a new adversarial erasing approach for localizing and expanding object regions progressively. Starting with a single small object region, our proposed approach drives the classification network to sequentially discover new and complement object regions by erasing the current mined regions in an adversarial manner. These localized regions eventually constitute a dense and complete object region for learning semantic segmentation. To further enhance the quality of the discovered regions by adversarial erasing, an online prohibitive segmentation learning approach is developed to collaborate with adversarial erasing by providing auxiliary segmentation supervision modulated by the more reliable classification scores. Despite its apparent simplicity, the proposed approach achieves 55.0% and 55.7% mean Intersection-over-Union (mIoU) scores on PASCAL VOC 2012 val and test sets, which are the new state-of-the-arts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Incremental Learning in Semantic Segmentation from Image Labels (2021, background): “Previous works focused on improving the pseudo-labels through multiple refinements steps [2, 3], additional losses [5, 10, 31, 34, 59, 63], or erasing techniques that force the CAM to expand and focus on non-discriminative parts of the image [12, 29, 64].”
SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning (2021, background): “For example, the weakly-supervised semantic segmentation approaches [29, 31, 14] generate pseudo-labels filtering out the background regions using saliency map, and recent data augmentations [17, 16] also exploit the saliency information to find the optimal mixing of mask.”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “There exist many region-wise segmentation methods [1,2,21,20,64,50,2,60] that organize the pixels into a set of regions (usually super-pixels), and then classify each region to get the image segmentation result.”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, methodology): “Subsequently, many approaches have achieved high accuracy using only image-level labels [9, 10, 40, 41], bounding boxes [42, 11, 12], scribbles [20] and points [13].”
Res2Net: A New Multi-Scale Backbone Architecture (2019, background): “Such ability of precisely localizing CAM region makes the Res2Net be potentially valuable for object region mining in weakly supervised semantic segmentation tasks [42].”
Stacked Autoencoders Driven by Semi-Supervised Learning for Building Extraction from near Infrared Remote Sensing Imagery (2021, methodology): “To derive a semantic segmentation, classification schemes can be applied [5].”
Conditional Generative Adversarial Network for Structured Domain Adaptation (2018, background): “Despite the existence of weakly-supervised semantic segmentation [42, 36], an enormous amount of laborintensive work is often required to annotate many images in order to achieve state-of-the-art semantic segmentation accuracy.”
Salient Object Detection in the Deep Learning Era: An In-Depth Survey (2019, neutral): “In computer vision, sampled applications of SOD include image understanding [1], [2], image cap-tioning [3]–[5], object detection [6], [7], un-supervised video object segmentation [8], [9], semantic segmentation [10]– [12], person re-identiﬁcation [13], [14], etc .”
Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing (2018, methodology): “AE-PSL [33] and DCSP [4] achieve the best performance among the baselines.”
Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground (2018, background): “The merit of SOD lies in it applications in many other computer vision tasks including: visual tracking [4], image retrieval [14,16], computer graphics [9], content aware image resizing [45], and weakly supervised semantic segmentation [18,39,40].”
Simultaneously Localize, Segment and Rank the Camouflaged Objects (2021, background): “[63] introduced the class activation map (CAM) to estimate the discriminative region of each class, which is the basis of many weakly supervised methods [1, 51, 17, 50, 24, 43, 47].”
Saliency Guided Self-Attention Network for Weakly and Semi-Supervised Semantic Segmentation (2019, methodology): “Therefore, a great amount of effort has been devoted to recover dense and reliable seeds [12], [13], [14], [15], [16].”
Understanding Deep Networks via Extremal Perturbations and Smooth Masks (2019, background): “Other works have leveraged perturbations at the input [23, 29] and intermediate layers [28] to perform weakly and fully supervised localization.”
Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation (2020, positive): “Adversarial erasing [15, 32] is a popular CAM expansion method, which erases the most discriminative part of CAM, guides the network to learn clas-siﬁcation features from other regions and expands activations.”
PANet: Few-Shot Image Semantic Segmentation With Prototype Alignment (2019, background): “Semi- and weaklysupervised learning methods [26, 3, 9, 15] alleviate such requirements but still need many weakly annotated training images.”
Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency (2019, background): “Attempting to alleviate the problem, several methods exploit weaker forms of supervision: image-level labels [1], [32], [38], bounding boxes [7], [29], or scribbles [20], [36].”
SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation (2018, background): “image-level labels [25, 23, 24], scribbles [15, 21], bounding boxes [13, 6] and points [2].”
Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup (2020, methodology): “The saliency information has been used in various ﬁelds of machine learning (Ren et al., 2013; Wei et al., 2017).”