Context Encoding for Semantic Segmentation

By Hang Zhang, Kristin J. Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, Amit Agrawal. 2018

1,368 citations recorded by the source corpus.

Recent work has made significant progress in improving spatial resolution for pixelwise labeling with Fully Convolutional Network (FCN) framework by employing Dilated/Atrous convolution, utilizing multi-scale features and refining boundaries. In this paper, we explore the impact of global contextual information in semantic segmentation by introducing the Context Encoding Module, which captures the semantic context of scenes and selectively highlights class-dependent featuremaps. The proposed Context Encoding Module significantly improves semantic segmentation results with only marginal extra computation cost over FCN. Our approach has achieved new state-of-the-art results 51.7% mIoU on PASCAL-Context, 85.9% mIoU on PASCAL VOC 2012. Our single model achieves a final score of 0.5567 on ADE20K test set, which surpasses the winning entry of COCO-Place Challenge 2017. In addition, we also explore how the Context Encoding Module can improve the feature representation of relatively shallow networks for the image classification on CIFAR-10 dataset. Our 14 layer network has achieved an error rate of 3.45%, which is comparable with state-of-the-art approaches with over 10Ã— more layers. The source code for the complete system are publicly available1.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “Among the approaches, most of methods [40, 41, 42, 20, 33, 39] adopt the ResNet-101 as backbone and ReﬁneNet [22] adopts a more powerful network, i.e ., ResNet-152, as the backbone.”
Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, methodology): “We follow the practice in Zhang [31] to choose training hyper-parameters.”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, background): “It also drives research progress in multimodal learning and inference across vision and language, which is one of the most active research areas in recent years [20, 18, 36, 19, 41, 4, 30, 5, 1, 31, 33, 32]”
K-Net: Towards Unified Image Segmentation (2021, background): “Based on this framework, many studies focus on enhancing the feature representation through dilated convolution [7–9], pyramid pooling [57, 67], context representations [64, 65], and attention mechanisms [32, 63, 68].”
Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation (2018, methodology): “In computer vision, incorporating context information has been shown to improve performance in various tasks including object recognition [12, 22, 18] and action recognition [9], object detection [3] and segmentation [25].”
Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation (2022, methodology): “CNN-based methods [23, 29, 50] utilize convolution operations to extract semantic information from feature maps for pixel-level label prediction.”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “Recently, deep neural networks (DNNs) became the dominant paradigm in various real-world applications, for example, image classiﬁcation [12, 18, 17, 16, 19], semantic segmentation [43, 32, 27], object detection [39, 15, 38, 46, 35].”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Examples of recent works exploiting contextual information include attention mechanisms [69, 76, 22, 30, 68, 61, 73], and fixed-scale aggregation [75, 12, 11, 72].”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, background): “The DPC architecture outperforms previous state-of-the-art models [95, 93] by more than 1.7%, and is comparable to concurrent works [14, 96, 48].”
Dual Attention Network for Scene Segmentation (2018, methodology): “Following [4,27], we employ a poly learning rate policy where the initial learning rate is multiplied by (1− iter total iter ) after each iteration.”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “We follow the standard testing procedure [32], [172].”
Object-Contextual Representations for Semantic Segmentation (2019, neutral): “37 - EncNet [75] ResNet-101 8× - - - 44.”
ResNeSt: Split-Attention Networks (2020, methodology): “For example, some methods add a pyramid module [8, 69] or introduce long-range connections [56] or use cross-channel feature-map attention [15, 65].”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, background): “It is worth mentioning a dual form of spatial attention, called channel-wise feature attention [37, 44, 21, 13].”
Attention-Guided Unified Network for Panoptic Segmentation (2018, background): “In the scope of scene understanding, [42] and [43] aggregated global contextual information as well as class-dependent features by channel-attention operations.”
YOLOv4: Optimal Speed and Accuracy of Object Detection (2020, methodology): “• Normalization of the network activations by their mean and variance: Batch Normalization (BN) [32], Cross-GPU Batch Normalization (CGBN or SyncBN) [93], Filter Response Normalization (FRN) [70], or Cross-Iteration Batch Normalization (CBN) [89]”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, methodology): “We use synchronized Batch Normalization in training as done in [60, 65].”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, background): “Additionally, EncNet [72] also adds supervision to their global feature.”