Geography-Aware Self-Supervised Learning

By Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David B. Lobell, Stefano Ermon. 2020

317 citations recorded by the source corpus.

Contrastive learning methods have significantly narrowed the gap between supervised and unsupervised learning on computer vision tasks. In this paper, we explore their application to geo-located datasets, e.g. remote sensing, where unlabeled data is often abundant but labeled data is scarce. We first show that due to their different characteristics, a non-trivial gap persists between contrastive and supervised learning on standard benchmarks. To close the gap, we propose novel training methods that exploit the spatio-temporal structure of remote sensing data. We leverage spatially aligned images over time to construct temporal positive pairs in contrastive learning and geo-location to design pre-text tasks. Our experiments show that our proposed method closes the gap between contrastive and supervised learning on image classification, object detection and semantic segmentation for remote sensing. Moreover, we demonstrate that the proposed method can also be applied to geo-tagged ImageNet images, improving downstream performance on various tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “absolute terms, it is necessary to notice once again how selfsupervision strategies lead to better results than exclusively supervised ones [12], [15], and, above all, how this is even more true when combining EO data from domains that are not homogeneous in terms of type of sensor, acquisition, res”
Self-Supervised Pretraining Improves Self-Supervised Pretraining (2021, background): “Furthermore, several recent papers that apply self-supervised learning to domain-specific problems did not apply transfer learning to the pretraining process itself, which motivated our work [57, 1, 29].”
Self-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Coding (2021, methodology): “Most of these applications either used small amounts, up to 50,000 images, of unlabeled training data [31, 30, 35] or tested learned representations on a small number of datasets [1, 19, 38].”
Towards Geospatial Foundation Models via Continual Pretraining (2023, methodology): “In recent years, a second approach has gained significant traction, where researchers aim to pretrained models specific to the geospatial domain [28, 2, 9, 38].”
SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (2023, background): “In summary, our technical contributions are: • We propose SkySense, the largest MM-RSFM to date with a modular design, which is capable of handling diverse tasks, from single-to multi-modal, static to temporal, and classification to localization.”
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution (2022, unknown): “We use a MoCo-TP (Ayush et al., 2020) checkpoint, pretrained on unlabeled FMoW satellite images.”
Geographical Knowledge-Driven Representation Learning for Remote Sensing Images (2021, background): “Recent researches have included geographical information in contrastive learning in order to achieve more efficient representation learning [24, 25].”
Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning (2022, methodology): “[3] leveraged the metadata from remote sensing images via spatially aligned but temporally separated images as positive pairs for contrastive learning and predicted the latitude and longitude as pretext tasks.”
An Empirical Study of Remote Sensing Pretraining (2022, background): “between positive samples, which are the unique characteristics in aerial scenes, while [52] simultaneously fuses the temporal information and geographical location into the MoCo-V2 [47] framework.”
Self-Supervised Learning in Remote Sensing: A review (2022, background): “Ayush et al. [161] designed a pretext task based on predicting the geo-location of the input image.”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “[209] proposed the geography-aware contrastive learning method to construct positive samples using spatially aligned images over time instead of different augmented views of the same image.”
Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data (2021, background): “[1] also propose to exploit the temporal information in satellite imagery to generate positive pairs and train a contrastive objective.”
RingMo: A Remote Sensing Foundation Model With Masked Image Modeling (2023, methodology): “…temporal information to acquire image pairs from the same location at different time points, which provides a natural enhancement to assist contrastive SSL. Ayush et al. [25] exploit spatially aligned images over time to build temporal positive pairs to design pretext tasks in contrastive learning.”
SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022, methodology): “Table 1 shows that compared to the previous stateof-the-art self-supervised method using a contrastive momentum encoding approach [30, 3], our SatMAE achieved a 6.”