An Empirical Study of Remote Sensing Pretraining

By Di Wang, Jing Zhang, Bo Du, Guisong Xia, Dacheng Tao. IEEE Transactions on Geoscience and Remote Sensing, 2022

301 citations recorded by the source corpus.

Deep learning has largely reshaped remote sensing (RS) research for aerial image understanding and made a great success. Nevertheless, most of the existing deep models are initialized with the ImageNet pretrained weights. Since natural images inevitably present a large domain gap relative to aerial images, probably limiting the finetuning performance on downstream aerial scene tasks. This issue motivates us to conduct an empirical study of remote sensing pretraining (RSP) on aerial images. To this end, we train different networks from scratch with the help of the largest RS scene recognition dataset up to now — MillionAID, to obtain a series of RS pretrained backbones, including both convolutional neural networks (CNN) and vision transformers such as Swin and ViTAE, which have shown promising performance on computer vision tasks. Then, we investigate the impact of RSP on representative downstream tasks including scene recognition, semantic segmentation, object detection, and change detection using these CNN and vision transformer backbones. Empirical study shows that RSP can help deliver distinctive performances in scene recognition tasks and in perceiving RS related semantics such as “Bridge” and “Airplane”. We also find that, although RSP mitigates the data discrepancies of traditional ImageNet pretraining on RS images, it may still suffer from task discrepancies, where downstream tasks require different representations from scene recognition tasks. These findings call for further research efforts on both large-scale pretraining datasets and effective pretraining methods. The codes and pretrained models will be released at https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing.

Subfields: Aerospace Engineering, Environmental Engineering, Geology

View Paper: https://www.semanticscholar.org/paper/6db718d6d6dfe94cb729e4dc9cdcc04e36d09631

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Fahad Shahbaz Khan
2. Dacheng Tao
3. Bo Du
4. Qiming Zhang
5. Jing Zhang
6. Jiwen Lu
7. Mu Li
8. Yi Zhu

In Environmental Engineering

1. Dacheng Tao
2. Jiwen Lu
3. Kun Fu
4. Xian Sun
5. Xingjian Shi
6. Guisong Xia

In Geology

1. Ji Qi
2. Dacheng Tao
3. Jiwen Lu
4. Haifeng Li
5. Qing Zhu
6. Chao Tao

Citations in the corpus

• RingMo: A Remote Sensing Foundation Model With Masked Image Modeling (2023, background): “For supervised learning [22], a large collection of labeled data is necessary to train the foundation model.”
• Towards Geospatial Foundation Models via Continual Pretraining (2023, background): “In recent years, a second approach has gained significant traction, where researchers aim to pretrained models specific to the geospatial domain [30, 2, 10, 40].”
• Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model (2022, methodology): “Most of the experimental settings are same with [10].”
• Transformers in Remote Sensing: A Survey (2022, methodology): “RSP [7] Classification, Segmentation, Detection MillionAID [33], Potsdam [70], iSAID [80], HRSC2016 [81], DOTA [49], CCD [82], LEVIR [61] Overall classification accuracy, mAP, F1 score Investigates pre-training transformers on a large-scale remote sensing dataset.”
• Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “pretrained ResNet50 model weights released by [260], since the sample labels of Million-AID are not publicly available.”