UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery

By Libo Wang, Rui Li, Ce Zhang, Shenghui Fang, Chenxi Duan, Xiaoliang Meng, Peter M. Atkinson. Isprs Journal of Photogrammetry and Remote Sensing, 2021

1,269 citations recorded by the source corpus.

Semantic segmentation of remotely sensed urban scene images is required in a wide range of practical applications, such as land cover mapping, urban change detection, environmental protection, and economic assessment.Driven by rapid developments in deep learning technologies, the convolutional neural network (CNN) has dominated semantic segmentation for many years. CNN adopts hierarchical feature representation, demonstrating strong capabilities for local information extraction. However, the local property of the convolution layer limits the network from capturing the global context. Recently, as a hot topic in the domain of computer vision, Transformer has demonstrated its great potential in global information modelling, boosting many vision-related tasks such as image classification, object detection, and particularly semantic segmentation. In this paper, we propose a Transformer-based decoder and construct a UNet-like Transformer (UNetFormer) for real-time urban scene segmentation. For efficient segmentation, the UNetFormer selects the lightweight ResNet18 as the encoder and develops an efficient global-local attention mechanism to model both global and local information in the decoder. Extensive experiments reveal that our method not only runs faster but also produces higher accuracy compared with state-of-the-art lightweight models. Specifically, the proposed UNetFormer achieved 67.8% and 52.4% mIoU on the UAVid and LoveDA datasets, respectively, while the inference speed can achieve up to 322.4 FPS with a 512x512 input on a single NVIDIA GTX 3090 GPU. In further exploration, the proposed Transformer-based decoder combined with a Swin Transformer encoder also achieves the state-of-the-art result (91.3% F1 and 84.1% mIoU) on the Vaihingen dataset. The source code will be freely available at https://github.com/WangLibo1995/GeoSeg.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/44e481c9645c057ea9273023aa7e23572226cee7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping (2022, methodology): “The U-NetFormer [54] selects the advanced ResNext101 as the encoder and develops an efficient global–local attention mechanism to model both global and local information in the decoder.”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “…benchmarked the performance of the proposed FTransUNet against twelve representative state-of-the-art methods, namely ABCNet [61], PSPNet [62], MAResU-Net [63], vFuseNet [15], FuseNet [14], ESANet [33], Tran-sUNet [31], SA-GATE [64], UNetFormer [44], CMFNet [19], MFTransNet [59], and CMGFNet [47].”
Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks (2023, background): “[9] introduced a recent breakthrough in the field, unveiling an efficient U-shaped transformer network custom-tailored for the precise execution of semantic segmentation tasks in VHR urban scene images.”
Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model (2022, neutral): “Therefore, it is unable to carry out an accurate pixel-level understanding as the latest proposed frameworks such as UNetFormer [79] and FactSeg [78].”
HeatPrompt: Zero-Shot Vision-Language Modeling of Urban Heat Demand from Satellite Images (2026, unknown): “However, converting pixel data into quantitative energy metrics continues to present significant challenges [4]–[6].”