CMTFNet: CNN and Multiscale Transformer Fusion Network for Remote-Sensing Image Semantic Segmentation

By Honglin Wu, Peng Huang, M. Zhang, Wenlong Tang, Xinyu Yu. IEEE Transactions on Geoscience and Remote Sensing, 2023

352 citations recorded by the source corpus.

Convolutional neural networks (CNNs) are powerful in extracting local information but lack the ability to model long-range dependencies. In contrast, the transformer relies on multihead self-attention mechanisms to effectively extract the global contextual information and thus model long-range dependencies. In this article, we propose a novel encoder–decoder structured semantic segmentation network, named CNN and multiscale transformer fusion network (CMTFNet), to extract and fuse local information and multiscale global contextual information of high-resolution remote-sensing images. Specifically, to further process the output features from the CNN encoder, we build a transformer decoder based on the multiscale multihead self-attention (M2SA) module for extracting rich multiscale global contextual information and channel information. Additionally, the transformer block introduces an efficient feed-forward network (E-FFN) to enhance the information interaction between different channels of the feature. Finally, the multiscale attention fusion (MAF) module fully fuses the feature information from different levels. We have conducted extensive comparison experiments and ablation experiments on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets. The extensive experimental results demonstrate that our proposed CMTFNet can obtain superior performance compared to the currently popular methods. The codes will be available at https://github.com/DrWuHonglin/CMTFNet.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/b06890b7d261ecf4b629608b63490ec1b74c4375

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Wanli Ouyang
2. Lei Bai
3. Sijie Zhao
4. Hao Chen
5. Pengfeng Xiao
6. Xue-liang Zhang

In Media Technology

1. Wanli Ouyang
2. Lei Bai
3. Sijie Zhao
4. Hao Chen
5. Pengfeng Xiao
6. Xue-liang Zhang

Citations in the corpus

• RS-Mamba for Large Remote Sensing Image Dense Prediction (2024, methodology): “…[19], PSPNet [42], HRNet [43], MA-FCN [44], Deeplabv3 + [45], ResUNet [46], MAP-Net [47], D-LinkNet [48], and SIINet [49], the compared transformer-based models include Segformer [50] and Road-Former [10], and the CNN-transformer hybrid models include BDTNet [51], TransUNet [52], and CMTFNet [53].”