Object-Contextual Representations for Semantic Segmentation

By Yuhui Yuan, Xilin Chen, Jingdong Wang. 2019

1,714 citations recorded by the source corpus.

In this paper, we address the semantic segmentation problem with a focus on the context aggregation strategy. Our motivation is that the label of a pixel is the category of the object that the pixel belongs to. We present a simple yet effective approach, object-contextual representations, characterizing a pixel by exploiting the representation of the corresponding object class. First, we learn object regions under the supervision of the ground-truth segmentation. Second, we compute the object region representation by aggregating the representations of the pixels lying in the object region. Last, % the representation similarity we compute the relation between each pixel and each object region, and augment the representation of each pixel with the object-contextual representation which is a weighted aggregation of all the object region representations according to their relations with the pixel. We empirically demonstrate that the proposed approach achieves competitive performance on various challenging semantic segmentation benchmarks: Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff. Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff. Our submission "HRNet + OCR + SegFix" achieves the 1-st place on the Cityscapes leaderboard by the time of submission. Code is available at: this https URL and this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Language-driven Semantic Segmentation (2022, neutral): “We compare the results to highly competitive standard semantic segmentation models, including OCNet (Yuan et al., 2020), ACNet (Fu et al., 2019), DeeplabV3 (Chen et al., 2017; Zhang et al., 2020a), and DPT (Ranftl et al., 2021).”
FocalClick: Towards Practical Interactive Image Segmentation (2022, methodology): “We select the current SOTA methods HRNet+OCR [38, 46] and SegFormer [42] as representatives.”
Semantic Segmentation in Art Paintings (2022, methodology): “Additionally, as HRNet+OCR [SZJ*19; YCW19] generalizes better than both DeepLab models over DRAMs unrealistic movements when trained on PASCAL VOC 2012, using it with domain adaptation may also provide an improvement in future results.”
Per-Pixel Classification is Not All You Need for Semantic Segmentation (2021, methodology): “C N N ba ck bo ne s OCRNet [50] R101c 520× 520 45.”
K-Net: Towards Unified Image Segmentation (2021, background): “Method Backbone Val mIoU OCRNet [64] HRNet-W48 44.”
ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, background): “Recent semantic segmentation models [12,37,62,63] have achieved state-of-the-art performance for recognizing general object/stuff classes from nat-ural scene images.”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, methodology): “All the HRNet networks are initialized with the ImageNet weights and supplied with the OCR module [11].”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, background): “attention module [9] and multiple scales [44], [45].”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Examples of recent works exploiting contextual information include attention mechanisms [69, 76, 22, 30, 68, 61, 73], and fixed-scale aggregation [75, 12, 11, 72].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “Currently, we have results (mIoU), which are depicted in Tables 3 4 5 6, by combining the HRNet with the object-contextual representation (OCR) scheme [170] 6 , a variant of object context [59], [171].”
Segmenter: Transformer for Semantic Segmentation (2021, methodology): “State-of-the-art methods deploy Fully Convolutional Networks (FCN) [44] and achieve impressive results on challenging segmentation benchmarks [10, 23, 57, 58, 60, 64, 66].”
Vision Transformers for Dense Prediction (2021, background): “Architectural research on dense prediction frequently focuses on the decoder and its aggregation strategy [6, 7, 52, 55].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, neutral): “9 HRNet-w48 [54] OCRNet [77] 71M 664G 45.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “Object-Contextual Representations (OCR) [42] learns object regions and the relation between each pixel and each object region, augmenting the representation pixels with the object-contextual representation.”