Weakly Supervised Instance Segmentation Using Class Peak Response

By Yanzhao Zhou, Yi Zhu, Qixiang Ye, Qiang Qiu, Jianbin Jiao. 2018

294 citations recorded by the source corpus.

Weakly supervised instance segmentation with image-level labels, instead of expensive pixel-level masks, remains unexplored. In this paper, we tackle this challenging problem by exploiting class peak responses to enable a classification network for instance mask extraction. With image labels supervision only, CNN classifiers in a fully convolutional manner can produce class response maps, which specify classification confidence at each image location. We observed that local maximums, i.e., peaks, in a class response map typically correspond to strong visual cues residing inside each instance. Motivated by this, we first design a process to stimulate peaks to emerge from a class response map. The emerged peaks are then back-propagated and effectively mapped to highly informative regions of each object instance, such as instance boundaries. We refer to the above maps generated from class peak responses as Peak Response Maps (PRMs). PRMs provide a fine-detailed instance-level representation, which allows instance masks to be extracted even with some off-the-shelf methods. To the best of our knowledge, we for the first time report results for the challenging image-level supervised instance segmentation task. Extensive experiments show that our method also boosts weakly supervised pointwise localization as well as semantic segmentation performance, and reports state-of-the-art results on popular benchmarks, including PASCAL VOC 2012 and MS COCO.

Subfields: Biophysics, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI (2019, methodology): “+ LRP on video deep learning, selective relevance method [51] 7 X + BiLRP [50] 7 X DeepLIFT [56] 7 X Prediction Difference Analysis [57] 7 X Slot Activation Vectors [40] 7 X PRM (Peak Response Mapping) [58] 7 X”
C-MIDN: Coupled Multiple Instance Detection Network With Segmentation Guidance for Weakly Supervised Object Detection (2019, background): “Weakly Supervised Semantic Segmentation A signiﬁ-cant advance of performance of WSSS has been witnessed in last several years [44, 16, 38, 1, 29, 45, 41].”
Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations (2019, methodology): “To resolve this issue, a recent approach [53] incorporates CAMs with an off-the-shelf segmentation proposal technique [2], which however has to be trained separately on an external dataset with additional supervision.”
Feature Pyramid Transformer (2020, result): “, GT and RT) capture more diverse and richer inter-object contexts to achieve better object recognition and detection performances, which is consistent with the conclusion of instance-level recognition works [45,46].”
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors (2019, methodology): “Recently, [37, 48, 23, 21] study instance segmentation algorithms that can generalize to categories without mask annotations.”
One-Shot Instance Segmentation (2018, background): “Weak supervision is a promising direction for annotation-heavy tasks, hence it has been explored for semantic segmentation [58, 57, 61, 17, 88, 7, 41], object detection [56, 91, 67] and instance segmentation [35, 33, 92].”
DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision (2021, background): “Here, the term “weakly supervised” can either refer to the relaxed supervision on bounding box location [85,86], or the absence of mask annotations [46–49].”
Large-Scale Interactive Object Segmentation With Human Annotators (2019, unknown): “For semantic/instance segmentation the following sources of supervision have been explored: image level label only [38], point clicks [3, 36], boxes only [7, 12, 35], scribbles only [18, 3], boxes plus clicks [4, 30, 35, 26, 22].”
Pointly-Supervised Instance Segmentation (2021, methodology): “With image-level labels, such methods rely on segmentation proposals [43,54] either re-ranking them [61] or generating pseudo-ground truth [1,2,62].”
LayerCAM: Exploring Hierarchical Class Activation Maps for Localization (2021, methodology): “Recently, PRM [35] utilizes local maximums, i.e., the peak values, as top signals to be back-propagated the network downwards in a winner-take-all manner to extract ﬁne-detailed class instance activation maps.”
Unsupervised Object Segmentation by Redrawing (2019, methodology): “PRM [59], instead, finds pixels that provoke peaks in saliency maps and uses these as a reference to choose the best regions out of a large set of proposals previously obtained using MCG [2], an unsupervised region proposal algorithm.”
Identity-Guided Human Semantic Parsing for Person Re-Identification (2020, background): “semantic segmentation [16, 47, 19, 12, 11, 33, 44, 62], which aim to locate objects like person, horse or dog at pixel-level with image-level supervision.”
Object Counting and Instance Segmentation With Image-Level Supervision (2019, methodology): “In addition to PRM, the image-level supervised object detection methods MELM [28], CAM [33] and SPN [35] used with MCG and reported by [34] are also included in Tab.”
Deep Learning Enables Accurate Diagnosis of Novel Coronavirus (COVID-19) With CT Images (2020, methodology): “More importantly, these methods required only image-level annotations and have been shown able to achieve close accuracy to models trained on refined expert annotations [24].”