Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation

By Jiwoon Ahn, Suha Kwak. 2018

898 citations recorded by the source corpus.

The deficiency of segmentation labels is one of the main obstacles to semantic segmentation in the wild. To alleviate this issue, we present a novel framework that generates segmentation labels of images given their image-level class labels. In this weakly supervised setting, trained models have been known to segment local discriminative parts rather than the entire object area. Our solution is to propagate such local responses to nearby areas which belong to the same semantic entity. To this end, we propose a Deep Neural Network (DNN) called AffinityNet that predicts semantic affinity between a pair of adjacent image coordinates. The semantic propagation is then realized by random walk with the affinities predicted by AffinityNet. More importantly, the supervision employed to train AffinityNet is given by the initial discriminative part segmentation, which is incomplete as a segmentation annotation but sufficient for learning semantic affinities within small image areas. Thus the entire framework relies only on image-level class labels and does not require any extra data or annotations. On the PASCAL VOC 2012 dataset, a DNN learned with segmentation labels generated by our method outperforms previous models trained with the same level of supervision, and is even as competitive as those relying on stronger supervision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, background): “In a more data-efﬁcient setting, weakly-supervised segmentation methods exploit annotations that are even easier to obtain, e.g. image-level tags [3,30,46].”
Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “Following the standard methodology [3, 34], we augment the Pascal VOC dataset with images from [26] for a total of 10582 images for training and 1449 for validation annotated on 20 object categories.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, background): “Another line of related research is of weaklysupervised semantic segmentation [1,9,26,31,36,40,65,68, 80], which aims to learn semantic segmentation with only image-level object category supervision.”
Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, methodology): “To this end, existing methods mainly focus on generating better seed areas [25, 57, 59, 58] and expanding these seed areas [1, 2, 20, 23, 55].”
Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency (2019, positive): “Attempting to alleviate the problem, several methods exploit weaker forms of supervision: image-level labels [1], [32], [38], bounding boxes [7], [29], or scribbles [20], [36].”
Learning Integral Objects With Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation (2020, background): “To the best of our knowledge, the previous best VGG16 based result without saliency is achieved by AfﬁnityNet [1] with mIoU 58.4% on the validation set.”
C-MIDN: Coupled Multiple Instance Detection Network With Segmentation Guidance for Weakly Supervised Object Detection (2019, methodology): “Without losing generality, we choose AfﬁnityNet [1], one of the state-of-the-art WSSS methods.”
Puzzle-CAM: Improved Localization Via Matching Partial And Full Features (2021, methodology): “Most previous methods [4, 5, 6] using WSSS are based on the class activation maps (CAMs) [7] to achieve good performance.”
Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations (2019, methodology): “7 AffinityNet [1] I 58.”
Weakly-Supervised Semantic Segmentation via Sub-Category Exploration (2020, methodology): “Table 1 shows that our approach significantly improves the IoU over AffinityNet [1] by almost 3% using”
Saliency Guided Self-Attention Network for Weakly and Semi-Supervised Semantic Segmentation (2019, background): “AffinityNet [40], DSRG [13], SGDN [22], and SSDD [23] adopt region growing techniques to expand seeds.”
Single-Stage Semantic Segmentation From Image Labels (2020, methodology): “We compute our final classification scores as yc ≡ ynGWP c + ysize-focal c and use the multi-label soft-margin loss function [39] used in previous work [3, 57] as the classification loss,”
Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation (2021, methodology): “Previous methods [2, 13, 41] generate pseudo-masks and refine them with the CRF post-processing algorithm [26] or affinity network [2].”
Pseudo-mask Matters in Weakly-supervised Semantic Segmentation (2021, methodology): “To reduce the annotation burden, weaklysupervised semantic segmentation approaches have been proposed using scribble annotations [23, 32], bounding boxes [36, 9, 16] , points [3] or image-level labels [17, 27, 28, 2, 33].”
Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation (2020, methodology): “, view the CAM-activated regions as initial “seeds” and gradually grow the seed regions until cover the complete objects [2, 22, 27, 62].”
LayerCAM: Exploring Hierarchical Class Activation Maps for Localization (2021, background): “…ability of the class activation maps can make up such an issue of image-level labels, which further facilitates the ill-posed weakly-supervised tasks, such as weakly-supervised semantic segmentation [4], [5], [6], [7] and weakly-supervised object localization [8], [9] under image-level supervision.”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “Many weakly object localization and weakly semantic segmentation methods [7, 89, 182] start from an interpretation, and obtain promising results.”