Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks

By Nicolas Audebert, Bertrand Le Saux, Sébastien Lefèvre. Asian Conference on Computer Vision, 2016

406 citations recorded by the source corpus.

This work investigates the use of deep fully convolutional neural networks (DFCNN) for pixel-wise scene labeling of Earth Observation images. Especially, we train a variant of the SegNet architecture on remote sensing data over an urban area and study different strategies for performing accurate semantic segmentation. Our contributions are the following: (1) we transfer efficiently a DFCNN from generic everyday images to remote sensing images; (2) we introduce a multi-kernel convolutional layer for fast aggregation of predictions at multiple scales; (3) we perform data fusion from heterogeneous sensors (optical and laser) using residual correction. Our framework improves state-of-the-art accuracy on the ISPRS Vaihingen 2D Semantic Labeling dataset.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Ocean Engineering

View Paper: https://www.semanticscholar.org/paper/e9de7cc307b6c2a39383ef1a9acbca1ea3dfe27c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

In Ocean Engineering

Citations in the corpus

More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification (2020, methodology): “[37] is to segment multimodal EO data—high-resolution RGB and DSM images—using a multi-scaled network design.”
A review of deep learning methods for semantic segmentation of remote sensing imagery (2021, background): “Hence, normalized cross-entropy over the entire patch is used (Audebert et al., 2016; Audebert et al., 2018).”
Deep learning in remote sensing: a review (2017, methodology): “The authors released the pretrained model to reproduce the results of [142] on the Vaihingen data set.”
ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data (2019, background): “It should be noted that several authors report performance gains when they use inputs viewed at di ﬀ erent scales, e.g. , Audebert et al. (2016) and Yang et al. (2018).”
Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection (2016, methodology): “The strongest competitors are INRIA (89.5%, Maggiori et al., 2016), using a variant of fcn, and ONERA (89.8%, Audebert et al., 2016), with a variant of segnet.”
Unstructured Point Cloud Semantic Labeling Using Deep Segmentation Networks (2017, methodology): “• Prediction fusion using residual correction [ALSL16] (ﬁgure 6e).”
Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models (2018, background): “, 2017), fully convolutional models (Audebert et al., 2016), deconvolutions (Volpi and Tuia, 2017), stacking activations (Maggiori et al.”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, background): “Audebert et al. (2016) further improved the state-of-the-art for dense scene labeling of aerial images by exploiting the encoderdecoder architecture of SegNet (Badrinarayanan et al. (2015)).”
Continual Learning for Dense Labeling of Satellite Images (2019, methodology): “F1-scores [11] of each method for all the classes are re-ported in Table 2.”
Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters (2018, methodology): “Dataset SegNet FCN CNN + RF Mulit-Scale Deep Network [33] CNN + RF + CRF Ours”
Deep Learning Meets SAR: Concepts, models, pitfalls, and perspectives (2020, neutral): “experimental results not outperforming the state of the art by the time [177] likely because of network design or training strategy, they deduced that introducing advanced models and simultaneously using both data sources can greatly improve the performance of semantic segmentation.”
Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks (2017, neutral): “This process was investigated in [2] where a residual correction module was introduced.”
DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images (2018, methodology): “Audebert et al. [18] trained a variant of the SegNet architecture with multi-kernel convolutional layer, and achieved 89.8% on Vaihingen.”
Multitask learning for large-scale semantic change detection (2018, methodology): “In particular, methods involving CNNs have recently been proposed, yielding good performances (Audebert et al., 2016).”
Deep Learning for Land Use and Land Cover Classification Based on Hyperspectral and Multispectral Earth Observation Data: A Review (2020, methodology): “Conditional random fields (CRFs) is the main technique used for this end [178], and it has been successfully practised by many studies jointly with deep learning models targeting semantically segmented maps [75,179,180].”
Self-Attention for Raw Optical Satellite Time Series Classification (2019, background): “Following these developments, mono-temporal Earth observation approaches have adapted 2D convolutional neural networks from computer vision with great success [35, 36, 37, 38].”
Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks (2023, background): “Further, the same authors extended their work in [14] by implementing the multi-scale deep fully convolutional networks (FCNs) [15, 16] based on SegNet [17] to process and understand multimodal RS data for land cover segmentation [18].”
Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data (2018, background): “2925416 In the last decade, with the great advances in deep neural networks, notably convolutional neural networks (CNNs), it has been possible to obtain accurate segmentations [1]–[3].”