Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform

By Liang-Chieh Chen, Jonathan T. Barron, George Papandreou, Kevin P. Murphy, Alan L. Yuille. 2015

382 citations recorded by the source corpus.

Deep convolutional neural networks (CNNs) are the backbone of state-of-art semantic image segmentation systems. Recent work has shown that complementing CNNs with fully-connected conditional random fields (CRFs) can significantly enhance their object localization accuracy, yet dense CRF inference is computationally expensive. We propose replacing the fully-connected CRF with domain transform (DT), a modern edge-preserving filtering method in which the amount of smoothing is controlled by a reference edge map. Domain transform filtering is several times faster than dense CRF inference and we show that it yields comparable semantic segmentation results, accurately capturing object boundaries. Importantly, our formulation allows learning the reference edge map from intermediate CNN features instead of using the image gradient magnitude as in standard DT filtering. This produces task-specific edges in an end-to-end trainable system optimizing the target semantic segmentation quality.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (2016, methodology): “In a different direction, [63] replace the bilateral filtering module used in mean field inference with a faster domain transform module [67], improving the speed and lowering the memory requirements of the overall system, while [18], [68] combine semantic segmentation with edge detection.”
Holistically-Nested Edge Detection (2015, background): “…(1) carefully designed and/or learned features [26, 5], (2) multi-scale response fusion [38, 30, 28], (3) engagement of different levels of visual perception [16, 25, 37, 15] such as mid-level Gestalt law information [7], (4) structural information [6] and context [36], (5) holistic image…”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, background): “2 for illustration) that exploit context information for semantic segmentation [26, 14, 54, 8, 85, 48, 57, 65, 78].”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, background): “U-net type architectures [29, 75, 76, 87], Dense CRF post-processing [14, 51, 113] or bilateral filtertype smoothing [12, 39], as well as more general structured prediction with CNN-based pairwise terms [11,61] - but we leave this for future work.”
Attention to Scale: Scale-Aware Semantic Image Segmentation (2015, methodology): “Recently, [10] has been shown that further improvement can be attained by combining our proposed model and a discriminatively trained domain transform [23].”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, background): “29 ]; introducing boundary information [30–37]; designing various attention modules [38–46]; or using AutoML technologies [47–51].”
Learning Affinity via Spatial Propagation Networks (2017, methodology): “(9) has been used previously for semantic segmentation [4] and for learning low-level vision filters [19].”
Laplacian Reconstruction and Refinement for Semantic Segmentation (2016, methodology): “These pairwise features have been successfully integrated with CNN unary class predictions [12,13,14], typically using a fully connected CRF [15] or closely related architectures such as domain transfer [11].”
ReSeg: A Recurrent Neural Network for Object Segmentation (2015, methodology): “Recently however a few approaches in this direction have been explored (see, e.g., Graves & Schmidhuber, 2009; Visin et al., 2015; Kalchbrenner et al., 2015; Chen et al., 2015).”
Learning to predict crisp boundaries (2018, background): “Nowadays, it is expected to localize semantically meaningful objects boundaries, which play a fundamental and signiﬁcant role in many computer vision tasks such as image segmentation [4,5,6,7] and optical ﬂow [8,9].”
Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection (2016, positive): “Later deeplab was extended to simultaneously learn edge detection and semantic segmentation (Chen et al., 2015).”
Learning Random-Walk Label Propagation for Weakly-Supervised Semantic Segmentation (2017, methodology): “More recently, [2] proposed a method to jointly learn semantic edges and a CNN-based semantic labeler in an gradient-based learning framework.”
Fast Semantic Image Segmentation with High Order Context and Guided Filtering (2016, methodology): “We quantitatively compare our proposed model with state-of-art models: Deeplab [30], CRF-RNN [7], Deeplab-DT [31], DPN [21] and Piecewise [8] The result of the comparison on Pascal VOC 2012 test set is shown in Table 3.”
Adaptive Affinity Field for Semantic Segmentation (2018, background): “Besides, low-level cues, such as affinity [25,26] and contour [27,28] have also been leveraged to encode image structures.”
Deep Joint Image Filtering (2016, positive): “Similar observations are also found in [35] where a reference edge map is learned first from intermediate CNN features for the semantic segmentation.”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “Since it was proposed, numerous efforts have been devoted to improving FCN, by, for example, enlarging the receptive field [15, 16, 25, 124, 128, 135]; strengthening context cues [4, 43, 47, 48, 56, 57, 70, 75, 77, 81, 90, 126, 128, 129, 132, 138, 141]; leveraging boundary information [6, 14, 27, 66”