Exploring Cross-Image Pixel Contrast for Semantic Segmentation

By Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, Luc Van Gool. 2021

586 citations recorded by the source corpus.

Current semantic segmentation methods focus only on mining "local" context, i.e., dependencies between pixels within individual images, by context-aggregation modules (e.g., dilated convolution, neural attention) or structure-aware optimization criteria (e.g., IoU-like loss). However, they ignore "global" context of the training data, i.e., rich semantic relations between pixels across different images. Inspired by recent advance in unsupervised contrastive representation learning, we propose a pixel-wise contrastive algorithm for semantic segmentation in the fully supervised setting. The core idea is to enforce pixel embeddings belonging to a same semantic class to be more similar than embeddings from different classes. It raises a pixel-wise metric learning paradigm for semantic segmentation, by explicitly exploring the structures of labeled pixels, which were rarely explored before. Our method can be effortlessly incorporated into existing segmentation frameworks without extra overhead during testing. We experimentally show that, with famous segmentation models (i.e., DeepLabV3, HRNet, OCR) and backbones (i.e., ResNet, HRNet), our method brings performance improvements across diverse datasets (i.e., Cityscapes, PASCAL-Context, COCO-Stuff, CamVid). We expect this work will encourage our community to rethink the current de facto training paradigm in semantic segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luc Van Gool
2. Elisa Ricci
3. Andrew J. Davison
4. Enrico Fini
5. Tianfei Zhou
6. Wenguan Wang
7. Moin Nabi
8. Qi Wu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Dacheng Tao
4. Ming-Hsuan Yang
5. Yi Yang
6. Ling Shao
7. Wenguan Wang
8. Chunhua Shen

Citations in the corpus

• Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, methodology): “Very recently, in the context of semantic segmentation, the advantage of contrastive pre-training was demonstrated in [53] and a pixel-wise contrastive loss has been proposed [46] showing promising results.”
• Bootstrapping Semantic Segmentation with Regional Contrast (2021, methodology): “In concurrent work, (Wang et al., 2021a; Alonso et al., 2021) also perform contrastive learning with active sampling.”
• Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation (2021, background): “It plays a vital role in many computer vision tasks, such as image editing and medical image analysis [32, 47].”
• Rethinking Semantic Segmentation: A Prototype View (2022, methodology): “In [62, 111], while cluster-/pixel-level metric loss is adopted to regularize representation, the pixel class is still inferred via parametric softmax.”
• Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels (2022, background): “In semantic segmentation, contrastive learning has become a promising new paradigm [1,29,41,48,50].”