SPGNet: Semantic Prediction Guidance for Scene Parsing

By Bowen Cheng, Liang-Chieh Chen, Yunchao Wei, Yukun Zhu, Zilong Huang, Jinjun Xiong, Thomas S. Huang, Wen-mei W. Hwu, Humphrey Shi. 2019

117 citations recorded by the source corpus.

Multi-scale context module and single-stage encoder-decoder structure are commonly employed for semantic segmentation. The multi-scale context module refers to the operations to aggregate feature responses from a large spatial extent, while the single-stage encoder-decoder structure encodes the high-level semantic information in the encoder path and recovers the boundary information in the decoder path. In contrast, multi-stage encoder-decoder networks have been widely used in human pose estimation and show superior performance than their single-stage counterpart. However, few efforts have been attempted to bring this effective design to semantic segmentation. In this work, we propose a Semantic Prediction Guidance (SPG) module which learns to re-weight the local features through the guidance from pixel-wise semantic prediction. We find that by carefully re-weighting features across stages, a two-stage encoder-decoder network coupled with our proposed SPG module can significantly outperform its one-stage counterpart with similar parameters and computations. Finally, we report experimental results on the semantic segmentation benchmark Cityscapes, in which our SPGNet attains 81.1% on the test set using only 'fine' annotations.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, neutral): “Unet [28], Deeplabv3+ [8], MSCI [21], SPGNet [2], ReﬁneNet [22] and DFN [36] adopted encoder-decoder structures that fuse the information in low-level and high-level layers to predict segmentation mask.”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, background): “Semantic segmentation was long tackled as a pixel classification problem with CNNs [5, 6, 8, 20, 32].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Besides this communication module, our MaX-DeepLab employs a stackedhourglass-style decoder [78, 71, 19].”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, neutral): “5% SPGNet [9] ICCV’19 ResNet-101 81.”
OCNet: Object Context Network for Scene Parsing (2018, neutral): “2 SPGNet (Cheng et al., 2019) 2× ResNet-50 3 81.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “Net) [128], Semantic Prediction Guidance for Scene Parsing (SPGNet) [129], gated shape CNNs [130], Adaptive Context”
Real-Time Scene Text Detection With Differentiable Binarization and Adaptive Scale Fusion (2022, background): “RefineNet [28], Deeplabv3+ [5], DFN [60], and SPGNet [6] adopted encoder-decoder structure which fuse high-level and low-level features to get more discriminating feature.”
Interactive Object Segmentation With Inside-Outside Guidance (2022, methodology): “O VER the past few years, we have witnessed a revolutionary advancement in semantic [9], [10], [11], [12], [13], [14], [15], [16], [17], [18] and instance segmentation [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29] for different domains, such as general scenes [3], [30], [31],…”
Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation (2020, unknown): “Besides exploiting various contextual information from the visual cues [25,11,14,12,5,43], depth data have recently been utilized as supplementary information to RGB data to achieve improved segmentation accuracy [26,33,44,4,15,23,6,19].”
Interactive Object Segmentation With Inside-Outside Guidance (2020, background): “The success of CNN-based interactive segmentation has beneﬁted sig-niﬁcantly from the development of FCNs. Speciﬁcally, FCN [44], DeepLab series [10, 11, 12] and PSP [68] have been directly applied to tackle the interactive segmentation [67, 38, 45, 46].”
Deep Interactive Thin Object Selection (2021, methodology): “In this work, we employ a ResNet-50 [14] variant of DeepLabv3+ [5] as the backbone, which is proven successful across various segmentation tasks [44, 6].”
Disentangled Non-Local Neural Networks (2020, unknown): “2 SPGNet [7] ResNet-101 81.”
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation (2019, background): “(1) Encoder-decoder [31, 16, 9, 37, 1, 25, 41, 10] captures the context information in the encoder path and recover high resolution features in the decoder path.”