Towards Geospatial Foundation Models via Continual Pretraining
By Mat'ias Mendieta, Boran Han, Xingjian Shi, Yi Zhu, Chen Chen, Mu Li. 2023
156 citations recorded by the source corpus.
Geospatial technologies are becoming increasingly essential in our world for a wide range of applications, including agriculture, urban planning, and disaster response. To help improve the applicability and performance of deep learning models on these geospatial tasks, various works have begun investigating foundation models for this domain. Researchers have explored two prominent approaches for introducing such models in geospatial applications, but both have drawbacks in terms of limited performance benefit or prohibitive training cost. Therefore, in this work, we propose a novel paradigm for building highly effective geospatial foundation models with minimal resource cost and carbon impact. We first construct a compact yet diverse dataset from multiple sources to promote feature diversity, which we term GeoPile. Then, we investigate the potential of continual pretraining from large-scale ImageNet-22k models and propose a multi-objective continual pretraining paradigm, which leverages the strong representations of ImageNet while simultaneously providing the freedom to learn valuable in-domain features. Our approach outperforms previous state-of-the-art geospatial pretraining methods in an extensive evaluation on seven downstream datasets covering various tasks such as change detection, classification, multi-label classification, semantic segmentation, and super-resolution. Code is available at https://github.com/mmendiet/GFM.
Subfields: Computer Vision and Pattern Recognition, Media Technology, Ocean Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Valerio Marsocci
- Simone Scardapane
- Yongjun Zhang
- Bo Dang
- Xin Guo
- Dingxiang Hu
- Jiangwei Lao
- Yingying Zhang
In Media Technology
Citations in the corpus
- Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “More recent works employed different datasets for their SSL solutions [13], [14], [15], but focused their effort toward multimodal models, based on existing CV approaches [16].”
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (2023, methodology): “SkySense incorporates 2.06 billion parameters and is pre-trained on a large-scale multi-modal dataset which comprises 21.5 million RSI temporal sequences extracted from high-spatial-resolution optical images (HSROIs), medium-resolution temporal multispectral imagery (TMsI) and temporal SAR imagery (”