Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation

By Zhi Tian, Tong He, Chunhua Shen, Youliang Yan. 2019

239 citations recorded by the source corpus.

Recent semantic segmentation methods exploit encoder-decoder architectures to produce the desired pixel-wise segmentation prediction. The last layer of the decoders is typically a bilinear upsampling procedure to recover the final pixel-wise prediction. We empirically show that this oversimple and data-independent bilinear upsampling may lead to sub-optimal results. In this work, we propose a data-dependent upsampling (DUpsampling) to replace bilinear, which takes advantages of the redundancy in the label space of semantic segmentation and is able to recover the pixel-wise prediction from low-resolution outputs of CNNs. The main advantage of the new upsampling layer lies in that with a relatively lower-resolution feature map such as 1/16 or 1/32 of the input size, we can achieve even better segmentation accuracy, significantly reducing computation complexity. This is made possible by 1) the new upsampling layer's much improved reconstruction capability; and more importantly 2) the DUpsampling based decoder's flexibility in leveraging almost arbitrary combinations of the CNN encoders' features. Experiments on PASCAL VOC demonstrate that with much less computation complexity, our decoder outperforms the state-of-the-art decoder. Finally, without any post-processing, the framework equipped with our proposed decoder achieves new state-of-the-art performance on two datasets: 88.1% mIOU on PASCAL VOC with 30% computation of the previously best model; and 52.5% mIOU on PASCAL Context.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

FCOS: Fully Convolutional One-Stage Object Detection (2019, background): “Recently, fully convolutional networks (FCNs) [20] have achieved tremendous success in dense prediction tasks such as semantic segmentation [20, 28, 9, 19], depth estimation [17, 31], keypoint detection [3] and counting [2].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Currently methods pay more attention to multi-scale feature aggregation [3, 34, 52,53,59,65,68,81].”
Object-Contextual Representations for Semantic Segmentation (2019, neutral): “7 - - - DUpsampling [58] Xception-71 4× M - - - - 52.”
Conditional Convolutions for Instance Segmentation (2020, background): “In computer vision, the closest task to instance segmentation is semantic segmentation, for which fully convolutional networks (FCNs) have shown dramatic success [8, 17, 26, 27, 36].”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, methodology): “For continuous feature space learning, prior work use multi-scale feature aggregation [37, 5, 33, 18, 42, 31, 32, 2, 44, 4, 48, 17] to fuse the contextual information by probing the incoming features with filters or pooling operations at multiple rates and multiple fields-of-view.”
OCNet: Object Context Network for Scene Parsing (2018, methodology): “The scaling factor d is used to to solve the small gradient problem of softmax function according to Vaswani et al. (2017) and we set d = C2 .”
FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation (2019, methodology): “Data-Dependent Upsampling DUpsampling [29] is also related to our method, which takes advantages of the redundancy in the segmentation label space and is able to recover the pixel-wise prediction from low-resolution outputs of CNNs. Compared to our method, DUpsampling has a strong dependency on the…”
Using labeled data to evaluate change detectors in a multivariate streaming environment (2009, unknown): “In order to achieve better segmentation effect, we adopt DUpsampling [29] instead of bilinear interpolation for upsampling.”
FCOS: A Simple and Strong Anchor-Free Object Detector (2020, background): “mantic segmentation [10], [11], [12], depth estimation [13], [14], keypoint detection [15] and counting.”
Adaptive Context Network for Scene Parsing (2019, methodology): “Among the approaches, the recent methods[21, 28] use more powerful network(e.”
EfficientPS: Efficient Panoptic Segmentation (2020, background): “Tian et al (2019) propose data-dependent upsampling which accounts for the redundancy in the label space as opposed to simple bilinear upsampling.”
Knowledge Adaptation for Efficient Semantic Segmentation (2019, background): “Semantic segmentation is a crucial and challenging task for image understanding [3, 5, 4, 33, 14, 33, 16, 17, 24].”
SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection (2020, methodology): “4: Examples of the experimental results on (a) the SYNTHIA road dataset and (b) our R2D road dataset: (i) RGB, (ii) Depth, (iii) SNE-Depth (Ours), (iv) RGBD and (v) SNE-RGBD (Ours); (1) DeepLabv3+ [9], (2) U-Net [26], (3) SegNet [3], (4) GSCNN [30], (5) DUpsampling [32], (6) DenseASPP [37], (7) Fuse”
Enforcing Geometric Constraints of Virtual Normal for Depth Prediction (2019, background): “Recently, owing to the structural features from very deep convolution neural network, such as ResNet [18], various pixel-wise vision tasks have been promoted significantly, such as semantic segmentation [42, 43, 19].”