Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation

By Xingwei He, Yong Zhou, Jiaqi Zhao, Di Zhang, Rui Yao, Yang Xue. IEEE Transactions on Geoscience and Remote Sensing, 2022

537 citations recorded by the source corpus.

Global context information is essential for the semantic segmentation of remote sensing (RS) images. However, most existing methods rely on a convolutional neural network (CNN), which is challenging to directly obtain the global context due to the locality of the convolution operation. Inspired by the Swin transformer with powerful global modeling capabilities, we propose a novel semantic segmentation framework for RS images called ST-U-shaped network (UNet), which embeds the Swin transformer into the classical CNN-based UNet. ST-UNet constitutes a novel dual encoder structure of the Swin transformer and CNN in parallel. First, we propose a spatial interaction module (SIM), which encodes spatial information in the Swin transformer block by establishing pixel-level correlation to enhance the feature representation ability of occluded objects. Second, we construct a feature compression module (FCM) to reduce the loss of detailed information and condense more small-scale features in patch token downsampling of the Swin transformer, which improves the segmentation accuracy of small-scale ground objects. Finally, as a bridge between dual encoders, a relational aggregation module (RAM) is designed to integrate global dependencies from the Swin transformer into the features from CNN hierarchically. Our ST-UNet brings significant improvement on the ISPRS-Vaihingen and Potsdam datasets, respectively. The code will be available at https://github.com/XinnHe/ST-UNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/31426bfcfbf477acc230bcb82d32e2b7e4e546eb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Man On Pun
2. Xiaokang Zhang
3. Xianping Ma
4. Xinyu Yu
5. Peng Huang
6. Wenlong Tang
7. Honglin Wu
8. Ming Liu

In Computer Vision and Pattern Recognition

1. Man On Pun
2. Xiaokang Zhang
3. Xianping Ma
4. Xinyu Yu
5. Peng Huang
6. Wenlong Tang
7. Honglin Wu
8. Ming Liu

In Media Technology

1. Man On Pun
2. Xiaokang Zhang
3. Xianping Ma
4. Ming Liu

Citations in the corpus

• CMTFNet: CNN and Multiscale Transformer Fusion Network for Remote-Sensing Image Semantic Segmentation (2023, methodology): “With the rapid development of transformers in the field of natural language processing (NLP), some researchers have tried to solve this problem by using transformers in semantic segmentation tasks [22], [23], [24], [25].”
• A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “Recently, deep learning (DL)-based methods have been successfully developed for remote sensing semantic segmentation by exploiting convolutional neural networks (CNNs) [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25].”