Understanding Convolution for Semantic Segmentation

By Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, Garrison W. Cottrell. 2017

1,908 citations recorded by the source corpus.

Recent advances in deep learning, especially deep convolutional neural networks (CNNs), have led to significant improvement over previous semantic segmentation systems. Here we show how to improve pixel-wise semantic segmentation by manipulating convolution-related operations that are of both theoretical and practical value. First, we design dense upsampling convolution (DUC) to generate pixel-level prediction, which is able to capture and decode more detailed information that is generally missing in bilinear upsampling. Second, we propose a hybrid dilated convolution (HDC) framework in the encoding phase. This framework 1) effectively enlarges the receptive fields (RF) of the network to aggregate global information; 2) alleviates what we call the "gridding issue"caused by the standard dilated convolution operation. We evaluate our approaches thoroughly on the Cityscapes dataset, and achieve a state-of-art result of 80.1% mIOU in the test set at the time of submission. We also have achieved state-of-theart overall on the KITTI road estimation benchmark and the PASCAL VOC2012 segmentation task. Our source code can be found at https://github.com/TuSimple/TuSimple-DUC.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “Most of methods [5, 22, 40, 27, 30, 41, 35, 19, 42, 36] adopt the same backbone as ours and the others [32, 34] utilize stronger backbones.”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, methodology): “Note that we do not employ the multi-grid method [75, 15, 10], which we found does not improve the performance.”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, methodology): “Motivated by multigrid methods which employ a hierarchy of grids of different sizes [4, 69, 5, 57] and following [72, 16], we adopt different atrous rates within block4 to block7 in the proposed model.”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, background): “This “stem” has been shown to be effective for segmentation in [87, 77].”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, methodology): “The work in [87] employs a hybrid of rates within the last two blocks of ResNet [34], while Deformable ConvNets [22] proposes the deformable convolution which generalizes atrous convolution by learning the rates.”
Dual Attention Network for Scene Segmentation (2018, neutral): “DUC [22] 77.”
In-place Activated BatchNorm for Memory-Optimized Training of DNNs (2017, methodology): “We did not apply hybrid dilated convolutions [27] nor added an auxiliary loss as proposed in [33].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, neutral): “DUC-HDC [144] 77.”
Dilated Residual Networks (2017, background): “Such artifacts are shown in Figure 3(c) and have also been observed in concurrent work on semantic segmentation [17].”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, methodology): “Multi-grid: Motivated by the success of employing a hierarchy of different atrous rates in semantic segmentation [71, 20, 12], we modify the atrous rates in (1) the last residual block shared for predicting both semantic and direction features, and (2) the block for box classifier.”
ExFuse: Enhancing Feature Fusion for Semantic Segmentation (2018, positive): “We use two public-available semantic segmentation benchmarks – PASCAL VOC 2012 [20] and Semantic Boundaries Dataset [29] – for training and evaluate performances on PASCAL VOC 2012 validation set, which is consistent with many previous work [1,2,3,4,5,6,8,9,10,13,11,12,27].”
Spherical CNNs on Unstructured Grids (2019, background): “…Segmentation Image semantic segmentation is a classic problem in computer vision, and there has been an impressive body of literature studying semantic segmentation of planar images (Ronneberger et al., 2015; Badrinarayanan et al., 2015; Long et al., 2015; J´egou et al., 2017; Wang et al., 2018a).”
OCNet: Object Context Network for Scene Parsing (2018, neutral): “1 DUC-HDC [16]‡ WACV2018 ResNet-101 77.”
BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation (2018, methodology): “DUC [32], PSPNet [40], DeepLab v2 [5], and Deeplab v3 [6] use the dilated convolution to preserve the spatial size of the feature map.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “include the DeepLab family [76], multiscale context aggregation [77], Dense Upsampling Convolution and Hybrid Dilated Convolution (DUC-HDC) [78], densely connected Atrous Spatial Pyramid Pooling (DenseASPP) [79], and the”
SDC – Stacked Dilated Convolution: A Unified Descriptor Network for Dense Matching Tasks (2019, background): “Yet, a sequence of dilated convolutions can introduce gridding effects (different output nodes use disjoint subsets of input nodes) if dilation rates are not selected properly [39].”
Gated Residual Networks with Dilated Convolutions for Supervised Speech Separation (2018, positive): “We assign the dilation rates following a sawtooth wave-like fashion [18]: a set of residual blocks are grouped together to form the “rising edge” of the wave which has exponentially increasing dilation rates, and two succeeding groups repeat the same pattern.”