Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model

By Di Wang, Qiming Zhang, Yufei Xu, Jing Zhang, Bo Du, Dacheng Tao, L. Zhang. IEEE Transactions on Geoscience and Remote Sensing, 2022

379 citations recorded by the source corpus.

Large-scale vision foundation models have made significant progress in visual tasks on natural images, with vision transformers (ViTs) being the primary choice due to their good scalability and representation ability. However, large-scale models in remote sensing (RS) have not yet been sufficiently explored. In this article, we resort to plain ViTs with about 100 million parameters and make the first attempt to propose large vision models tailored to RS tasks and investigate how such large models perform. To handle the large sizes and objects of arbitrary orientations in RS images, we propose a new rotated varied-size window attention to replace the original full attention in transformers, which can significantly reduce the computational cost and memory footprint while learning better object representation by extracting rich context from the generated diverse windows. Experiments on detection tasks show the superiority of our model over all state-of-the-art models, achieving 81.24% mean average precision (mAP) on the DOTA-V1.0 dataset. The results of our models on downstream classification and segmentation tasks also show competitive performance compared to existing advanced methods. Further experiments show the advantages of our models in terms of computational complexity and data efficiency in transferring. The code and models will be released at https://github.com/ViTAE-Transformer/Remote-Sensing-RVSA.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/8db0d69e5a6db578ffdd5a6518d43bb04c420be4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, methodology): “Several recent studies, including SatMAE [12], Scale-MAE [13], ViTAE [14], Billion-scale MAE [15], RingMo [16], and geospatial foundation models (GFM) [17], have employed MIM on large vision transformers (ViT) and large-scale satellite imagery datasets, yielding encouraging results.”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, background): “RS image interpretation is crucial for monitoring and analyzing the Earth’s surface and environment, with widespread applications in military defense, geological surveying, and urban planning [1], [2], [3].”
Large Selective Kernel Network for Remote Sensing Object Detection (2023, methodology): “Transformer-based [54] models, such as the Vision Transformer (ViT) [14, 49, 55, 11, 1], Swin transformer [36, 22, 63, 76, 47] and PVT [57] have gained popularity in computer vision due to their effectiveness in image recognition tasks.”
GeoChat:Grounded Large Vision-Language Model for Remote Sensing (2023, methodology): “To address this, we propose to utilize ViTAE-RVSA [31] model, pre-trained on the LoveDA dataset [32], which encompasses the required important classes.”
SpectralGPT: Spectral Remote Sensing Foundation Model (2023, background): “However, these advanced models have been relatively underexplored in RS. Wang et al. [26] trained a plain vision transformer with 100 million parameters on RGB images towards RS task design and developed a new rotated varied-size window attention mechanism for fine-tuning the model on downstream…”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “Ontheotherhand,RVSA[243]introduces a novel rotated varied-size window attention mechanism that advances the plain vision transformer to serve as a fundamental model for various remote sensing tasks.”