SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery

By Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, Stefano Ermon. 2022

602 citations recorded by the source corpus.

Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Developing similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information, we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneficial. Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to $\uparrow$ 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to $\uparrow$ 14%) and semantic segmentation. Code and data are available on the project website: https://sustainlab-group.github.io/SatMAE/

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, methodology): “More recent works employed different datasets for their SSL solutions [13], [14], [15], but focused their effort toward multimodal models, based on existing CV approaches [16].”
SpectralGPT: Spectral Remote Sensing Foundation Model (2023, methodology): “98.73 ViT-22k [22] ImageNet-22k 98.91 SatMAE [31] fMoW-S2 99.09 SpectralGPT fMoW-S2 99.15 SpectralGPT + fMoW-S2+BigEarthNet 99.”
Towards Geospatial Foundation Models via Continual Pretraining (2023, methodology): “Comparatively, the SOTA geospatial pretrained method SatMAE [9] requires 768 hours on a V100 GPU and 109.”
ClimaX: A foundation model for weather and climate (2023, unknown): “A few recent works have proposed pretraining techniques for satellite imagery and remote sensing (Yuan & Lin, 2020; Cong et al., 2022; Reed et al., 2022a) but they have so far not been applied to multi-sensory data and variables in weather and climate.”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “These include scale-invariance (scale-MAE [13]), temporal information (SatMAE [12]), and temporal invariance (SeCo [39]), among others.”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, methodology): “Our goal is to break down the differences between nat-ural images and RS images that lead to the obstacles to the application of MLLMs in the RS domain and propose a universal MLLM specifically customized for RS.”
Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning (2022, methodology): “We show that Scale-MAE leads to better performing, more robust multiscale representations than both a standard MAE and a recently proposed, state-of-the-art remote sensing MAE called SatMAE [13] across remote sensing datasets with a variety of scale and resolution characteristics.”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, methodology): “We have conducted experiments employing the following methods: ResNet50 pre-trained on ImageNet [65], ViT-B pretrained with MAE [81], SatMAE [95], Scale-MAE [96], SeCo [97], and CACo [98].”