Multimodal Fusion Transformer for Remote Sensing Image Classification

By Swalpa Kumar Roy, Ankur Deria, Danfeng Hong, Behnood Rasti, Antonio J. Plaza, Jocelyn Chanussot. IEEE Transactions on Geoscience and Remote Sensing, 2022

500 citations recorded by the source corpus.

Vision transformers (ViTs) have been trending in image classification tasks due to their promising performance when compared with convolutional neural networks (CNNs). As a result, many researchers have tried to incorporate ViTs in hyperspectral image (HSI) classification tasks. To achieve satisfactory performance, close to that of CNNs, transformers need fewer parameters. ViTs and other similar transformers use an external classification (CLS) token, which is randomly initialized and often fails to generalize well, whereas other sources of multimodal datasets, such as light detection and ranging (LiDAR), offer the potential to improve these models by means of a CLS. In this article, we introduce a new multimodal fusion transformer (MFT) network, which comprises a multihead cross-patch attention (mCrossPA) for HSI land-cover classification. Our mCrossPA utilizes other sources of complementary information in addition to the HSI in the transformer encoder to achieve better generalization. The concept of tokenization is used to generate CLS and HSI patch tokens, helping to learn a distinctive representation in a reduced and hierarchical feature space. Extensive experiments are carried out on widely used benchmark datasets, i.e., the University of Houston (UH), Trento, University of Southern Mississippi Gulfpark (MUUFL), and Augsburg. We compare the results of the proposed MFT model with other state-of-the-art transformers, classical CNNs, and conventional classifiers models. The superior performance achieved by the proposed model is due to the use of mCrossPA. The source code will be made available publicly at https://github.com/AnkurDeria/MFT.

Subfields: Ecology, Media Technology

View Paper: https://www.semanticscholar.org/paper/6f80235d415ad23d5a96aa652a2d1b3e7f6eb778

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Ecology

In Media Technology

Citations in the corpus

Transformers in Remote Sensing: A Survey (2022, background): “A Multi-modal fusion transformer, MFT, is introduced in [119], that comprises a data fusion scheme to derive class tokens in the transformers from multi-modal data (e.g., LiDAR, SAR) along with the standard hyperspectral patch tokens.”
Spectral–Spatial Morphological Attention Transformer for Hyperspectral Image Classification (2023, methodology): “Roy et al. [50] recently developed a multimodal fusion trans-former (MFT) to extract features from HSIs and fuse them with a CLS token derived from light detection and ranging (LiDAR) data to enhance the joint classification performance.”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “However, in this line of research, existing works normally utilize single-level feature fusion based on summation or concatenation and ignore long-range cross-modal dependencies at different feature levels [38].”
Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework (2023, background): “Xue et al. [18] and Roy et al. [41] have also made their first attempts to fuse HS and LiDAR data by means of modifying transformers, respectively.”