Multi-class Token Transformer for Weakly Supervised Semantic Segmentation

By Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaïd, Dan Xu. Computer Vision and Pattern Recognition, 2022

302 citations recorded by the source corpus.

This paper proposes a new transformer-based framework to learn class-specific object localization maps as pseudo labels for weakly supervised semantic segmentation (WSSS). Inspired by the fact that the attended regions of the one-class token in the standard vision transformer can be leveraged to form a class-agnostic localization map, we investigate if the transformer model can also effectively capture class-specific attention for more discriminative object localization by learning multiple class tokens within the transformer. To this end, we propose a Multi-class Token Transformer, termed as MCTformer, which uses multiple class tokens to learn interactions between the class tokens and the patch tokens. The proposed MCTformer can successfully produce class-discriminative object localization maps from the class-to-patch attentions corresponding to different class tokens. We also propose to use a patch-level pairwise affinity, which is extracted from the patch-to-patch transformer attention, to further refine the localization maps. Moreover, the proposed framework is shown to fully complement the Class Activation Mapping (CAM) method, leading to remarkably superior WSSS results on the PASCAL VOC and MS COCO datasets. These results underline the importance of the class token for WSSS. 11https://github.com/xulianuwa/MCTformer

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/772f9f21511de7bc1077b877a0ea0bd6e50f4e76

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring (2022, background): “…model the global contexts and achieves signiﬁcant progress in lots of high-level vision tasks (e.g., image classiﬁcation [11], object detection [1,31] and semantic segmentation [25,32]), it has been developed to solve image super-resolution [10], image deblurring [21,27] and image denoise [2,24].”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “For weakly supervised semantic segmentation, previous works [253], [303] improve the typical CNN pipeline with class activation maps (CAM) and use refined CAM as training labels, which requires an extra model for training.”