The color out of space: learning self-supervised representations for Earth Observation imagery

By Stefano Vincenzi, Angelo Porrello, Pietro Buzzega, Marco Cipriano, Pietro Fronte, Roberto Cuccu, Carla Ippoliti, Annamaria Conte, Simone Calderara. 2020

65 citations recorded by the source corpus.

The recent growth in the number of satellite images fosters the development of effective deep-learning techniques for Remote Sensing (RS). However, their full potential is untapped due to the lack of large annotated datasets. Such a problem is usually countered by fine-tuning a feature extractor that is previously trained on the ImageNet dataset. Unfortunately, the domain of natural images differs from the RS one, which hinders the final performance. In this work, we propose to learn meaningful representations from satellite imagery, leveraging its high-dimensionality spectral bands to reconstruct the visible colors. We conduct experiments on land cover classification (BigEarthNet) and West Nile Virus detection, showing that colorization is a solid pretext task for training a feature extractor. Furthermore, we qualitatively observe that guesses based on natural images and colorization rely on different parts of the input. This paves the way to an ensemble model that eventually outperforms both the above-mentioned techniques.

Subfields: Atmospheric Science, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atmospheric Science

1. Zhenwei Shi
2. Guisong Xia
3. Bo Du
4. Dacheng Tao
5. Jing Zhang
6. Di Wang
7. Lichao Mou
8. Vladimir Risojevic

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Simone Scardapane
3. Valerio Marsocci
4. Bo Du
5. Jing Zhang
6. Di Wang
7. Guisong Xia
8. Zhenwei Shi

In Media Technology

1. Simone Scardapane
2. Valerio Marsocci
3. Zhenwei Shi
4. Bo Du
5. Guisong Xia
6. Dacheng Tao
7. Jing Zhang
8. Di Wang

Citations in the corpus

• Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “[18] learns visual representations inferring information on the visible spectrum from the other bands on BigEarthNet [19].”
• Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification (2020, background): “Vincenzi et al. attempted to predict visible bands of remote sensing images by other spectral bands [52].”
• An Empirical Study of Remote Sensing Pretraining (2022, background): “In this case, it is promising to develop self-supervised pretraining methods [46]–[49] and some related methods have been developed in the remote sensing area [50]–[53].”
• Towards Geospatial Foundation Models via Continual Pretraining (2023, methodology): “[37] employs a colorization objective on Sentinel-2 imagery utilizing the various spectral bands.”
• Self-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Coding (2021, methodology): “Most of these applications either used small amounts, up to 50,000 images, of unlabeled training data [31, 30, 35] or tested learned representations on a small number of datasets [1, 19, 38].”
• Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, background): “[139] suggested training the model by predicting color channels based on multispectral channels, not including RGB instead of gray-scale channels, to fully exploit the diverse features of each band.”
• Self-Supervised Learning in Remote Sensing: A review (2022, background): “[135] provided an initial attempt to leverage spectral context for SSL (Figure 7).”
• Geographical Knowledge-Driven Representation Learning for Remote Sensing Images (2021, methodology): “[22] propose using high-dimensional data to reconstruct image color for pre-training, which they believe will aid networks in learning image representations.”