Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade

By Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, Xiaoou Tang. 2017

279 citations recorded by the source corpus.

We propose a novel deep layer cascade (LC) method to improve the accuracy and speed of semantic segmentation. Unlike the conventional model cascade (MC) that is composed of multiple independent models, LC treats a single deep model as a cascade of several sub-models. Earlier sub-models are trained to handle easy and confident regions, and they progressively feed-forward harder regions to the next sub-model for processing. Convolutions are only calculated on these regions to reduce computations. The proposed method possesses several advantages. First, LC classifies most of the easy regions in the shallow stage and makes deeper stage focuses on a few hard regions. Such an adaptive and difficulty-aware learning improves segmentation performance. Second, LC accelerates both training and testing of deep network thanks to early decisions in the shallow stage. Third, in comparison to MC, LC is an end-to-end trainable framework, allowing joint learning of all sub-models. We evaluate our method on PASCAL VOC and Cityscapes datasets, achieving state-of-the-art performance and fast speed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “tions, such as image classification [12], [32], object detection [33] and semantic segmentation [34].”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, neutral): “…VOC [76] 79.1 LRR 4x ResNet-CRF [21] 79.3 DeepLabv2-CRF [10] 79.7 CentraleSupelec Deep G-CRF [7] 80.2 HikSeg COCO [71] 81.4 Deep Layer Cascade (LC) [45] 82.7 TuSimple [75] 83.1 Large Kernel Matters [60] 83.6 Multipath-ReﬁneNet [47] 84.2 ResNet-38 MS COCO [77] 84.9 PSPNet [84] 85.4 IDW-CNN [74]…”
Hybrid Task Cascade for Instance Segmentation (2019, background): “This task is closely related to both object detection and semantic segmentation [30, 22].”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, methodology): “Finally, our proposed Semantic Prediction Guidance (SPG) bears a similarity to the Layer Cascade method [34] which treats each pixel differently.”
BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation (2018, methodology): “Recently, the algorithms [1,17,25,39] of real-time semantic segmentation have shown that there are mainly three approaches to accelerate the model.”
Region Proposal by Guided Anchoring (2019, methodology): “Since there is no need to consider those excluded regions, we replace the ensuing convolutional layers by masked convolution [17, 28] for more efficient inference.”
Open Compound Domain Adaptation (2019, methodology): “Unlike existing curriculum adaptation methods [56, 6, 29, 25, 58, 57] that rely on some holistic measure of instance difficulty, we schedule the learning of unlabeled instances in the compound target domain according to their individual gaps to the labeled source domain, so that we solve an incremen”
Recent progress in semantic image segmentation (2018, methodology): “Apart frommulti-level strategy, amulti-stagemethod is used in semantic segmentation (Li et al. 2017b).”
Improving Semantic Segmentation via Video Propagation and Label Relaxation (2018, background): “Most of the hardest pixels to classify lie on the boundary between object classes [25].”
Adaptive Affinity Field for Semantic Segmentation (2018, background): “Since the introduction of fully convolutional networks [2], both deeper [4,18,19] and wider [20,21,22] network architectures have been proposed and have dramatically boosted the performance on standard benchmarks like PASCAL VOC 2012 [16].”
S2DNAS: Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search (2019, background): “They attempted to add extra controllers to select which computations are executed [5, 3, 34, 25, 15, 7, 46, 41, 42].”
Anytime Recognition with Routing Convolutional Networks (2019, methodology): “[ 24] treats a single deep network as a cascade of several sub-models and uses earlier sub-models to handle easy regions in semantic segmentation.”
ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning (2020, background): “This comes with the issue that predictions are especially uncertain close to the mixing borders, as the segmentation task is hardest close to class boundaries [25].”
Efficient Segmentation: Learning Downsampling Near Semantic Boundaries (2019, background): “Other complementary approaches include skipping some layers at some pixels [17] and early stopping of network computation for some spatial regions of the image [16, 33].”
Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform (2018, background): “The users were requested to rank 4 versions of each image: SRCNN [7], MemNet [44] (the state-of-the-art PSNR-oriented method), our SFT-GAN, and the original HR image according to their visual quality.”