Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection

By Dimitrios Marmanis, Konrad Schindler, Jan Dirk Wegner, Silvano Galliani, Mihai Datcu, Uwe Stilla. 2016

665 citations recorded by the source corpus.

We present an end-to-end trainable deep convolutional neural network (DCNN) for semantic segmentation with built-in awareness of semantically meaningful boundaries. Semantic segmentation is a fundamental remote sensing task, and most state-of-the-art methods rely on DCNNs as their workhorse. A major reason for their success is that deep networks learn to accumulate contextual information over very large windows (receptive fields). However, this success comes at a cost, since the associated loss of effecive spatial resolution washes out high-frequency details and leads to blurry object boundaries. Here, we propose to counter this effect by combining semantic segmentation with semantically informed edge detection, thus making class-boundaries explicit in the model, First, we construct a comparatively simple, memory-efficient model by adding boundary detection to the Segnet encoder-decoder architecture. Second, we also include boundary detection in FCN-type models and set up a high-end classifier ensemble. We show that boundary detection significantly improves semantic segmentation with CNNs. Our high-end ensemble achieves > 90% overall accuracy on the ISPRS Vaihingen benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Global and Planetary Change

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Global and Planetary Change

Citations in the corpus

ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “The visualization result implies that the ScaleNet focuses more on the edge information and specific semantic features, which efficiently improves image classification tasks [1,34].”
Look at Boundary: A Boundary-Aware Face Alignment Algorithm (2018, methodology): “It has been proved efficient in vision tasks as segmentation [32, 27, 22] and object detection [36, 50, 37].”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “They use their previous work Marmanis et al. (2016b) which uses an ensemble of fully convolutional networks to obtain pixelwise classification at full resolution of aerial images. Marmanis et al. (2016a) propose to compensate the loss of spatial resolution due to the pooling layers by combining sema”
Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation (2022, methodology): “[42] explicitly introduced the edge detection module [43] in a semantic segmentation network to supervise the boundary feature learning.”
Semantic segmentation of slums in satellite images using transfer learning on fully convolutional neural networks (2019, positive): “Expanding the three channel input limitations of traditional deep learning algorithms (Kemker et al., 2018; Marmanis et al., 2018) use specific architectures to use elevation information and multispectral imagery to boost performance in semantic segmentation frameworks.”
UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery (2021, neutral): “The encoder-decoder CNN-based methods, although have achieved encouraging performance, encounter bottlenecks in urban scene interpretation (Sherrah, 2016) (Marmanis et al., 2018; Nogueira et al., 2019).”
Deep learning in remote sensing: a review (2017, background): “In [143], the authors present a model that learns a CNN to separately output edge likelihoods at multiple scales from CIR and height data.”
Gated Convolutional Neural Network for Semantic Segmentation in High-Resolution Images (2017, background): “[28] embedded boundary detection to the SegNet encoder-decoder architecture.”
Change detection techniques for remote sensing applications: a survey (2019, background): “(Marmanis et al. 2018) suggested a Deep Convolution Neural Network (DCNN) for semantic segmentation in high resolution satellite images.”
Topological Map Extraction From Overhead Images (2018, background): “Today, most building footprint delineation from a single image is often approached via semantic segmentation as part of a broader multi-class task and many works exist, e.g., [40, 24, 30, 51, 20, 31].”
Multi-Task Learning for Segmentation of Building Footprints with Deep Neural Networks (2017, background): “This is often achieved on the architecture level by introducing special boundary refinement modules [13], [15], on the fusion level by combining feature maps with boundary predictions [4] or by using a different output representation in the training [16], [17].”
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images (2020, background): “Speciﬁcally, given a feature X = [ x 1 , x 2 , ..., x N ] ∈ R N × C , both the dot-attention and linear attention generate the query matrix Q , key matrix K and value matrix V .”
Very Deep Convolutional Neural Networks for Complex Land Cover Mapping Using Multispectral Remote Sensing Imagery (2018, background): “Importantly, CNN has led to a series of breakthroughs in several remote sensing applications, such as classification [15], segmentation [24], and object detection [25], due to its superior performance in a variety of applications relative to shallow-structured machine learning tools.”
A review of deep learning methods for semantic segmentation of remote sensing imagery (2021, methodology): “Similarly, Marmanis et al. (2018) extended the SegNet encoder-decoder architecture by adding boundary detection, which makes object boundaries explicit in the form of pixel-wise contour likelihood in the model.”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “Despite their good performance, CNN-based methods are ineffective in extracting global semantic information and long-range dependencies of input images due to the limited receptive fields of CNN [40], [41].”
Improving Semantic Segmentation via Video Propagation and Label Relaxation (2018, background): “Boundary handling Some prior works [12, 29] explicitly incorporate edge cues as constraints to handle boundary pixels.”