Gated-SCNN: Gated Shape CNNs for Semantic Segmentation

By Towaki Takikawa, David Acuna, Varun Jampani, Sanja Fidler. IEEE International Conference on Computer Vision, 2019

741 citations recorded by the source corpus.

Current state-of-the-art methods for image segmentation form a dense image representation where the color, shape and texture information are all processed together inside a deep CNN. This however may not be ideal as they contain very different type of information relevant for recognition. Here, we propose a new two-stream CNN architecture for semantic segmentation that explicitly wires shape information as a separate processing branch, i.e. shape stream, that processes information in parallel to the classical stream. Key to this architecture is a new type of gates that connect the intermediate layers of the two streams. Specifically, we use the higher-level activations in the classical stream to gate the lower-level activations in the shape stream, effectively removing noise and helping the shape stream to only focus on processing the relevant boundary-related information. This enables us to use a very shallow architecture for the shape stream that operates on the image-level resolution. Our experiments show that this leads to a highly effective architecture that produces sharper predictions around object boundaries and significantly boosts performance on thinner and smaller objects. Our method achieves state-of-the-art performance on the Cityscapes benchmark, in terms of both mask (mIoU) and boundary (F-score) quality, improving by 2% and 4% over strong baselines.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f5268324962fea5d4ee4596da7e183b9172fe0ea

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection (2020, background): “4: Examples of the experimental results on (a) the SYNTHIA road dataset and (b) our R2D road dataset: (i) RGB, (ii) Depth, (iii) SNE-Depth (Ours), (iv) RGBD and (v) SNE-RGBD (Ours); (1) DeepLabv3+ [9], (2) U-Net [26], (3) SegNet [3], (4) GSCNN [30], (5) DUpsampling [32], (6) DenseASPP [37], (7) Fuse”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, neutral): “1 GSCNN [35] WideResNet38 - - - - - - 80.”
Scaling Wide Residual Networks for Panoptic Segmentation (2020, background): “Speciﬁcally, the Wide-ResNet-38 (WR-38) [92], reﬁned by several human-crafted networks with extensive experiments, has been the de facto network back-bone for semantic segmentation [63, 73, 105, 82, 52], and instance segmentation [55, 32] on Cityscapes leader-board [14].”
Deep Interactive Thin Object Selection (2021, methodology): “[42] proposed a new gating mechanism that encourages the shape stream to only focus on processing boundary-relevant information.”
PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers (2022, methodology): “S and B denote semantic and boundary, and Add and Up refer to element-wise summation and bilinear Upsampling operation, respectively; BASLoss represents the boundary-awareness CE loss [46].”
Object-Contextual Representations for Semantic Segmentation (2019, neutral): “0 - - - GSCNN† [56] WideResNet-38 + ASPP 8× M 82.”
Semantic Edge Detection with Diverse Deep Supervision (2018, positive): “Gated-SCNN (Takikawa et al., 2019) converts the semantic edge representation from different ResNet layers to a representation suitable for segmentation, improving semantic segmentation substantially.”
Improving Semantic Segmentation via Decoupled Body and Edge Supervision (2020, methodology): “convolution network (GCN) [15,16] and dynamic graph [17]. For the second problem, prior arts manage to embed the low-level features that contain boundary and edge information into high-level features [18,19,20,21] or directly rene the outputs [22]. However, the interaction between the object body an”
Image Segmentation Using Deep Learning: A Survey (2020, background): “CNNs were initially proposed by Fukushima [16] in his seminal paper on the “Neocognitron”, which was based on Hubel and Wiesel’s hierarchical receptive field model of the visual cortex.”
Feature Pyramid Transformer (2020, neutral): “e ResNet-101 65.3 40.9 42.7 62.2 CFNet [19] ResNet-101 80.6 44.9 54.6 84.2 AFNB [33] ResNet-101 81.3 45.2 - - HRNet [47] HRNetV2-W48 81.6 44.7 55.984.5 OCNet [32] ResNet-101 81.7 45.5 54.7 84.3 GSCNN [48] Wide-ResNet-101 82.8 - 55.2 - PPM [15]+OC [32] ResNet-101 79.9 43.7 53.0 82.9 ASPP [14]+OC [32]”
SegFix: Model-Agnostic Boundary Refinement for Segmentation (2020, methodology): “Most of the existing works mainly addressed semantic segmentation through (i) increasing the resolution of feature maps [12,13,51], (ii) constructing more reliable context information [61,59,22,26,60] and (iii) exploiting boundary information [5,9,41,52].”
Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey (2020, methodology): “SCAL VOC 2012 Cityscapes [124] 82.2% 76.9% PSPNet [90] 2017 PASCAL VOC 2012 Cityscapes 85.4% 80.2% FC-DenseNet103 [115] 2017 CamVid road scene segmentation Gatech[125] 66.9% 79.4% Gated-SCNN [94] 2019 Cityscapes 82.8% for classication, detection and even for segmentation. In 2014, Hariharan et ”
Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN (2023, background): “vision-based tasks, including image classification [13]–[15], semantic segmentation [18]–[21], and object detection [22]– [25].”
Multiscale Deep Equilibrium Models (2020, neutral): “5 GSCNN [53] ResNet-50 - 73.”
Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision (2021, methodology): “Improving the segmentation quality on the edge areas include Gated-SCNN [31], PointRend [20], and SegFix [39].”
Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms (2021, methodology): “To further improve the performance, UPerNet [24] tries to identify many visual concepts, such as objects and textures in parallel; DUpsampling [25] exploits a data-dependent decoder that considers the correlation among the prediction of each pixel; and GSCNN [26] utilizes a novel architecture consis”
RELLIS-3D Dataset: Data, Benchmarks and Analysis (2020, methodology): “Gated-SCNN[33] is a two-stream CNN architecture for semantic segmentation that explicitly processes shape information in a separate branch, yielding a classical segmentation stream and a parallel shape stream.”