Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning

By Colorado Reed, Ritwik Gupta, Shufan Li, S. Brockman, Christopher Funk, Brian Clipp, Salvatore Candido, M. Uyttendaele, Trevor Darrell. IEEE International Conference on Computer Vision, 2022

417 citations recorded by the source corpus.

Large, pretrained models are commonly finetuned with imagery that is heavily augmented to mimic different conditions and scales, with the resulting models used for various tasks with imagery from a range of spatial scales. Such models overlook scale-specific information in the data for scale-dependent domains, such as remote sensing. In this paper, we present Scale-MAE, a pretraining method that explicitly learns relationships between data at different, known scales throughout the pretraining process. Scale-MAE pre-trains a network by masking an input image at a known input scale, where the area of the Earth covered by the image determines the scale of the ViT positional encoding, not the image resolution. Scale-MAE encodes the masked image with a standard ViT backbone, and then decodes the masked image through a bandpass filter to reconstruct low/high frequency images at lower/higher scales. We find that tasking the network with reconstructing both low/high frequency images leads to robust multiscale representations for remote sensing imagery. Scale-MAE achieves an average of a 2.4 − 5.6% non-parametric kNN classification improvement across eight remote sensing datasets compared to current state-of-the-art and obtains a 0.9 mIoU to 1.7 mIoU improvement on the SpaceNet building segmentation transfer task for a range of evaluation scales.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d3d079e711308bd278c270b621dde66aa311625c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “These include scale-invariance (scale-MAE [13]), temporal information (SatMAE [12]), and temporal invariance (SeCo [39]), among others.”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, background): “Image interpretation tasks include scene classification, image captioning, VQA, visual grounding (VG), and object detection.”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, background): “Compared with alternative meth-ods, RSPrompter produces superior results, characterized by sharper edges, more defined contours, improved completeness, and a closer approximation to the ground-truth references.”
ClimaX: A foundation model for weather and climate (2023, neutral): “…supervision bottleneck by pretraining (Devlin et al., 2018; He et al., 2022) large unsupervised “foundation” models (Bommasani et al., 2021) on huge passive datasets, such as text and images scraped from the internet (Ramesh et al., 2022; Brown et al., 2020; Liu et al., 2021; Reed et al., 2022b).”
Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar (2023, background): “More recently, also leveraging vision transformers, Reed et al. (2022) scale the Masked Auto-Encoder approach of He et al. (2022) and apply it to building segmentation.”
SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (2023, methodology): “We further compare our method equipping Swin-L with the two representative RSFMs: SatMAE [19] and Scale-MAE [57].”