More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification

By Danfeng Hong, Lianru Gao, Naoto Yokoya, Jing Yao, Jocelyn Chanussot, Qian Du, Bing Zhang. 2020

1,193 citations recorded by the source corpus.

Classification and identification of the materials lying over or beneath the earth’s surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS), and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML)—cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on “what,” “where,” and “how” to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS data sets. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Danfeng Hong
2. Jocelyn Chanussot
3. Antonio J. Plaza
4. Qian Du
5. Jing Yao
6. Swalpa Kumar Roy
7. Bing Zhang
8. Wei Li

In Computer Vision and Pattern Recognition

1. Danfeng Hong
2. Jocelyn Chanussot
3. Swalpa Kumar Roy
4. Antonio J. Plaza
5. Jing Yao
6. Qian Du
7. Xin Wu
8. Bing Zhang

In Media Technology

1. Jocelyn Chanussot
2. Danfeng Hong
3. Antonio J. Plaza
4. Swalpa Kumar Roy
5. Qian Du
6. Jing Yao
7. Bing Zhang
8. Xin Wu

In Multimodal Learning

1. Danfeng Hong
2. Jocelyn Chanussot

Citations in the corpus

• 3D Instance Segmentation of MVS Buildings (2021, background): “to overcome the challenge of information diversity [37].”
• Hyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects (2021, positive): “A single feature is insufficient to represent the whole image information, hence a combination of these features is used for image classification [47], [88], [98], [100]–[107].”
• Deep Learning for Unmanned Aerial Vehicle-Based Object Detection and Tracking: A survey (2021, background): “These data, either in content or time, are extremely informative, and, thus, object detection and tracking have entered the era of mass UAVs [1], [2], which have played an increasingly important role in land cover mapping [4], [5]; smart agriculture [6], [7]; smart cities [8]; traffic monitoring [6]”
• Multimodal Fusion Transformer for Remote Sensing Image Classification (2022, background): “Very recently, Hong et al. first proposed a general and unified multimodal deep learning framework for RS image classification [32] which provides an effective solution with respect to multi-source and multimodal RS products.”
• Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks (2023, methodology): “These aforementioned methods can be unified into a general multimodal deep learning framework for RS image land cover classification (i.e., MDL-RS) on both individual and cross-region environments [41].”
• A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, background): “Compared with natural images, high-resolution remote sensing images have more severe spectral heterogeneity and more complex spatial structures [3], [35].”
• Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework (2023, background): “Fortunately, the settings of cross-modal learning commonly allow us with a limited amount of multimodal training sources [30].”
• Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification (2020, methodology): “We compared the performance of different evaluated algorithms with the following metrics derived from the confusion matrix [18]: 1) Overall Accuracy (OA): This metric represents the proportion of correctly classified samples in all tested samples, and is computed by dividing the number of correctly…”
• SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery (2022, background): “Consequently, with respect to object detection, its capability to recognize objects on the Earth’s surface remains insufficient due to the deficiency of complementary information between different modalities [14].”
• UIU-Net: U-Net in U-Net for Infrared Small Object Detection (2022, methodology): “In recent years, the explosion of data-driven machine learning-based methods [16], [17], especially deep learning methods, has rapidly made them the most widely used method for detecting small infrared objects.”
• Deep learning for change detection in remote sensing: a review (2022, background): “…(Bazi et al. 2019), Unet (Jiao et al. 2020), Graph Convolutional Network (GCN) (Hong et al. 2020a; Gao et al. 2021), Spectral Former (Hong et al. 2021), and multimodal deep learning framework (Hong et al. 2020), which have been extensively used for remote sensing, are the potential models for DLCD.”
• SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers (2021, methodology): “3) Houston2013 Data: The third dataset was acquired by the ITRES CASI-1500 sensor over the campus of the University of Houston and its neighboring rural areas, Texas, USA, which, as a benchmark dataset, has been widely used for evaluating the performance of land cover classification [32].”
• Spectral–Spatial Morphological Attention Transformer for Hyperspectral Image Classification (2023, neutral): “More recently, a unified and general DL framework was developed by Hong et al. [28] for classification of RS imagery.”
• Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review (2022, methodology): “A representative DL-based method tackling this practical problem is proposed by (Hong et al., 2020), where a limited number of HS-MS or HS-SAR pairs are used in the training phase to realize a large-scale classification task in an area only covered by one modality data, i.e., MS or SAR.”