Causal Intervention for Weakly-Supervised Semantic Segmentation

By Dong Zhang, Hanwang Zhang, Jinhui Tang, Xiansheng Hua, Qianru Sun. 2020

548 citations recorded by the source corpus.

We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS). Specifically, we aim to generate better pixel-level pseudo-masks by using only image-level labels -- the most crucial step in WSSS. We attribute the cause of the ambiguous boundaries of pseudo-masks to the confounding context, e.g., the correct image-level classification of "horse" and "person" may be not only due to the recognition of each instance, but also their co-occurrence context, making the model inspection (e.g., CAM) hard to distinguish between the boundaries. Inspired by this, we propose a structural causal model to analyze the causalities among images, contexts, and class labels. Based on it, we develop a new method: Context Adjustment (CONTA), to remove the confounding bias in image-level classification and thus provide better pseudo-masks as ground-truth for the subsequent segmentation model. On PASCAL VOC 2012 and MS-COCO, we show that CONTA boosts various popular WSSS methods to new state-of-the-arts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hanwang Zhang
2. Xiansheng Hua
3. Kaihua Tang
4. Chunyan Miao
5. Xinting Hu
6. Qianru Sun
7. Yulei Niu
8. Ji-rong Wen

In Computer Vision and Pattern Recognition

1. Hanwang Zhang
2. Xiansheng Hua
3. Kaihua Tang
4. Qianru Sun
5. Chunyan Miao
6. Yulei Niu
7. Xinting Hu
8. Bernt Schiele

Citations in the corpus

• Distilling Causal Effect of Data in Class-Incremental Learning (2021, methodology): “It has been recently introduced to various computer vision fields, including feature learning [3, 46], few-shot classification [49], long-tailed recognition [40], semantic segmentation [8] and other high-level vision tasks[41, 32, 48, 27].”
• Interventional Few-Shot Learning (2020, background): “[67] proposes a retrospective for image captioning and other applications include image classification [10, 36], imitation learning [13], long-tailed recognition [56] and semantic segmentation [73].”
• Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation (2021, methodology): “[62], MCIS [44], IRN [1], FickleNet[30], SSDD [42], SEAM [49], SCE [7], CONTA [60].”
• Multi-class Token Transformer for Weakly Supervised Semantic Segmentation (2022, unknown): “…EPS (CVPR21) [23] ResNet101 ResNet38 I 62.7 64.3 SEAM (CVPR20) [39] ResNet38 I 64.5 65.7 BES (ECCV20) [5] ResNet101 I 65.7 66.6 CONTA (NeurIPS20) [48] ResNet38 I 66.1 66.7 AdvCAM (CVPR21) [22] ResNet101 I 68.1 68.0 ECS-Net (ICCV21) [ class token in DeiT-S to initialize the proposed multiple…”
• Counterfactual VQA: A Cause-Effect Look at Language Bias (2020, background): “Counterfactual thinking and causal inference have inspired several studies in computer vision, including visual explanations [20, 7, 42, 48], scene graph generation [12, 38], image recognition [38], video analysis [16, 26], zero-shot and few-shot learning [50, 49] incremental learning [22], represen”