Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation

By Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, Xilin Chen. 2020

776 citations recorded by the source corpus.

Image-level weakly supervised semantic segmentation is a challenging problem that has been deeply studied in recent years. Most of advanced solutions exploit class activation map (CAM). However, CAMs can hardly serve as the object mask due to the gap between full and weak supervisions. In this paper, we propose a self-supervised equivariant attention mechanism (SEAM) to discover additional supervision and narrow the gap. Our method is based on the observation that equivariance is an implicit constraint in fully supervised semantic segmentation, whose pixel-level labels take the same spatial transformation as the input images during data augmentation. However, this constraint is lost on the CAMs trained by image-level supervision. Therefore, we propose consistency regularization on predicted CAMs from various transformed images to provide self-supervision for network learning. Moreover, we propose a pixel correlation module (PCM), which exploits context appearance information and refines the prediction of current pixel by its similar neighbors, leading to further improvement on CAMs consistency. Extensive experiments on PASCAL VOC 2012 dataset demonstrate our method outperforms state-of-the-art methods using the same level of supervision. The code is released online.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, methodology): “These methods typically utilize Class Activation Maps (CAM) [64] to select the most discriminative regions, which are later used as pixel-level supervision for segmentation networks [10,21,56].”
Incremental Learning in Semantic Segmentation from Image Labels (2021, methodology): “Moreover, we show that also in this scenario From left to right: image, CAM, SEAM [63], SS [5], EPS [36], WILSON and the ground-truth.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, background): “Another line of related research is of weaklysupervised semantic segmentation [1,9,26,31,36,40,65,68, 80], which aims to learn semantic segmentation with only image-level object category supervision.”
A Survey on Vision Transformer (2020, background): “PSANet [151], OCNet [139], DANet [38] and CFNet [147] are the first works that introduce the self-attention module to semantic segmentation task, which consider and augment the relation and similarity [146, 74, 46, 89, 130] between the contextual pixels.”
Multi-class Token Transformer for Weakly Supervised Semantic Segmentation (2022, methodology): “We compared the model complexity of the proposed MCTformer with a commonly used CNN model for generating object localization maps [1, 39, 49], i.e ., ResNet38 [43], in terms of the number of parameters and multiply-add calculations (MACs).”
Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation (2020, positive): “During training, the equivariant attention [67] is also adopted.”
Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation (2021, background): “The recent work of SEAM [49] proposed a self-supervised equivariant attention mechanism to provide additional supervision for network learning.”
SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022, background): “In recent years, self-supervised learning techniques have quickly become the norm for pre-training models on large-scale natural image datasets [1, 2, 3, 4, 5, 6, 7, 8], and have demonstrated strong performance on downstream tasks including image classification [3, 4, 9, 10], image segmentation [3, ”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “For weakly supervised semantic segmentation, previous works [305], [306] improve the typical CNN pipeline with class activation maps (CAM) and use reﬁned CAM as training labels, which requires an extra model for training.”
Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation (2021, methodology): “Interest-ingly, SEAM shows a high confusion ratio and even worse than CAM.”
Causal Intervention for Weakly-Supervised Semantic Segmentation (2020, methodology): “To this end, existing methods mainly focus on generating better seed areas [25, 57, 59, 58] and expanding these seed areas [1, 2, 20, 23, 55].”
Open-Vocabulary Image Segmentation (2021, background): “Weakly-supervised semantic segmentation can train with image-level labels [23,29,34,45,49], of which reﬁning CAMs [53] is a popular techniques.”
Puzzle-CAM: Improved Localization Via Matching Partial And Full Features (2021, methodology): “SEAM [5] aims to refine CAMs using a pixel correlation module that captures context appearance information for each pixel and alters the original CAMs by using learned affinity attention maps.”
PseudoSeg: Designing Pseudo Labels for Semantic Segmentation (2020, methodology): “, 2018), utilizing pixel similarity to propagate the initial score map (Ahn & Kwak, 2018; Wang et al., 2020), or mining and co-segment the same category of objects across images (Sun et al.”
XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanations (2021, neutral): “As in general object localization [33], since an afﬁne transformation of an image does not change the relative location of a sign of the disease, it should not affect the occurrence map, either.”
Hypercorrelation Squeeze for Few-Shot Segmenation (2021, background): “To cope with the challenge, there have been various attempts in semi- and weakly-supervised segmentation approaches [6, 26, 39, 66, 72, 77, 88] which in turn effectively alleviated the data-hunger issue.”
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels (2021, background): “Semantic segmentation with less supervisions: Weakly-supervised semantic segmentation trains with image-level labels [36,47,24,31,52], of which refining CAMs [60] is a popular techniques.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Restrictions apply. image segmentation [181], [182], monocular vision [183], person reidentiﬁcation (re-ID) [184], [185], visual odometry [186], scene ﬂow estimation [187], knowledge distillation [188], optical ﬂow prediction [189], vision-language navigation [190], physiological signal estimation…”
Pseudo-mask Matters in Weakly-supervised Semantic Segmentation (2021, methodology): “To reduce the annotation burden, weaklysupervised semantic segmentation approaches have been proposed using scribble annotations [23, 32], bounding boxes [36, 9, 16] , points [3] or image-level labels [17, 27, 28, 2, 33].”