SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers

By Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio J. Plaza, Jocelyn Chanussot. 2021

1,401 citations recorded by the source corpus.

Hyperspectral (HS) images are characterized by approximately contiguous spectral information, enabling the fine identification of materials by capturing subtle spectral discrepancies. Due to their excellent locally contextual modeling ability, convolutional neural networks (CNNs) have been proven to be a powerful feature extractor in HS image classification. However, CNNs fail to mine and represent the sequence attributes of spectral signatures well due to the limitations of their inherent network backbone. To solve this issue, we rethink HS image classification from a sequential perspective with transformers and propose a novel backbone network called SpectralFormer. Beyond bandwise representations in classic transformers, SpectralFormer is capable of learning spectrally local sequence information from neighboring bands of HS images, yielding groupwise spectral embeddings. More significantly, to reduce the possibility of losing valuable information in the layerwise propagation process, we devise a cross-layer skip connection to convey memory-like components from shallow to deep layers by adaptively learning to fuse “soft” residuals across layers. It is worth noting that the proposed SpectralFormer is a highly flexible backbone network, which can be applicable to both pixelwise and patchwise inputs. We evaluate the classification performance of the proposed SpectralFormer on three HS datasets by conducting extensive experiments, showing the superiority over classic transformers and achieving a significant improvement in comparison with state-of-the-art backbone networks. The codes of this work will be available at https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of reproducibility.

Subfields: Atmospheric Science, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

3D Instance Segmentation of MVS Buildings (2021, background): “learns locally spectral representations from multiple neighboring bands instead of single bands [39].”
UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery (2021, positive): “…applied the Transformer for remote sensing image scene classification (Bazi et al., 2021; Deng et al., 2021), hyperspectral image classification (Hong et al., 2021) (He et al., 2021; Zhong et al., 2021), object detection (Li et al., 2022a), change detection (Chen et al., 2021a), building and…”
Extended Vision Transformer (ExViT) for Land Use and Land Cover Classification: A Multimodal Deep Learning Framework (2023, background): “The PW convolution is also necessary for adaptively combining related information along the channel dimension, as the role of groupwise spectral embedding module plays in SpectralFormer [17], however avoiding using 3-D convolution that tends to be hard to train with limited labelings.”
Transformers in Remote Sensing: A Survey (2022, methodology): “A transformers-based backbone, called SpectralFormer, is introduced in [8] that can take pixel-wise or patch-”
Spectral–Spatial Feature Tokenization Transformer for Hyperspectral Image Classification (2022, methodology): “More recently, Hong et al. [59] developed a new model called SpectralFormer (SF), which can learn spectral representation information from groupwise neighboring bands and construct a cross-layer transformer encoder (TE) module.”
Hyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects (2021, background): “Deep learning (DL) models is one of such revolutionary advancements in machine learning that improved HSIC accuracy [66]–[68].”
Deep learning for change detection in remote sensing: a review (2022, background): “Therefore, an immediate need for future studies is to incorporate AlexNet, ResNet, GoogleNet, Unet, GCN, Spectral Former, and multimodal deep learning framework algorithms in more DLCD applications.”
Multimodal Fusion Transformer for Remote Sensing Image Classification (2022, neutral): “We also considered a few state-of-the-art transformer models, i.e., ViT [49] and Spectral-This Former [22].”
MambaHSI: Spatial–Spectral Mamba for Hyperspectral Image Classification (2025, methodology): “SpectralFormer [16] learns sequence information from local signatures with a transformer.”