SEMANTIC SEGMENTATION OF AERIAL IMAGES WITH AN ENSEMBLE OF CNNS

By Dimitrios Marmanis, Jan Dirk Wegner, Silvano Galliani, Konrad Schindler, Mihai Datcu, Uwe Stilla. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2016

334 citations recorded by the source corpus.

This paper describes a deep learning approach to semantic segmentation of very high resolution (aerial) images. Deep neural architectures hold the promise of end-to-end learning from raw images, making heuristic feature design obsolete. Over the last decade this idea has seen a revival, and in recent years deep convolutional neural networks (CNNs) have emerged as the method of choice for a range of image interpretation tasks like visual recognition and object detection. Still, standard CNNs do not lend themselves to per-pixel semantic segmentation, mainly because one of their fundamental principles is to gradually aggregate information over larger and larger image regions, making it hard to disentangle contributions from different pixels. Very recently two extensions of the CNN framework have made it possible to trace the semantic information back to a precise pixel position: deconvolutional network layers undo the spatial downsampling, and Fully Convolution Networks (FCNs) modify the fully connected classification layers of the network in such a way that the location of individual activations remains explicit. We design a FCN which takes as input intensity and range data and, with the help of aggressive deconvolution and recycling of early network layers, converts them into a pixelwise classification at full resolution. We discuss design choices and intricacies of such a network, and demonstrate that an ensemble of several networks achieves excellent results on challenging data such as the <i>ISPRS semantic labeling benchmark</i>, using only the raw data as input.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8e0b61ca27a306e9232702986de293522e5599de

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection (2016, methodology): “For comparison, we also run our earlier dlr/eth model Marmanis et al. (2016), which is a combination of the fcn and vgg models that we will later also use in our ensemble, without explicit class-boundary detection.”
A survey of the recent architectures of deep convolutional neural networks (2019, background): “• Ensemble learning is one of the prospective areas of research in CNNs (Marmanis et al. 2016; Ahmed et al. 2019).”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “Marmanis et al. (2016a) propose to compensate the loss of spatial resolution due to the pooling layers by combining semantic segmentation with edge detection.”
Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters (2018, background): “There has been some research that tries to improve the performance of semantic segmentation and develop a deep neural network structure either by adding skip connections so as to reintroduce the high-frequency detailed information of an imagery after upsampling [34,35] or by using dilated convolutio”
Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning (2017, methodology): “…have used DCNNs pre-trained on ImageNet to yield state-of-the-art results for the semantic segmentation of highresolution multispectral aerial imagery (Wang et al., 2017; Marmanis et al., 2016) because the most widely used benchmarks (Rottensteiner et al., 2012) only use a single non-RGB band.”
Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models (2018, methodology): “Since since using the DSM has been shown to improve the segmentation results (Audebert et al. (2017); Marmanis et al. (2016); Volpi and Tuia (2017)) we use it in all of our experiments.”
Classification for High Resolution Remote Sensing Imagery Using a Fully Convolutional Network (2017, methodology): “…following the idea of Liang-Chieh Chen et al. [30], we use “atrous” convolution (also known as “dilation” convolution in other studies) instead of the ordinary convolution to increase the feature density, and build the multi-scale classification model by adding the skip-layer network architecture.”
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images (2020, background): “image), plays a crucial role in a wide range of applications such as land resources management, yield estimation and economic assessment [1-6].”
Learning Aerial Image Segmentation From Online Maps (2017, methodology): “They have already been successfully applied to semantic segmentation of aerial images, [e.g., 43–45].”
Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks (2016, methodology): “IRRG data “SVL”[23] RF + CRF[35] “DLR” (FCN)[22] Ours (SegNet)”
Deep Learning for Classification of Hyperspectral Data: A Comparative Review (2019, neutral): “In particular, standard optical imagery (Red-Green-Blue –RGB– and Infra-Red –IR–) beneﬁted from using deep convolutional neural networks (CNN) for tasks such as classiﬁcation, object detection or semantic segmentation [1], [2], [3].”
Topological Map Extraction From Overhead Images (2018, background): “Today, most building footprint delineation from a single image is often approached via semantic segmentation as part of a broader multi-class task and many works exist, e.g., [40, 24, 30, 51, 20, 31].”
A review of deep learning methods for semantic segmentation of remote sensing imagery (2021, methodology): “The encoder network in SegNet is structured contains a number of convolution and max pooling operations to extract features, which follows the architecture of FCNs.”