Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification

By Yuan Yuan, Lei Lin. 2020

177 citations recorded by the source corpus.

Satellite image time series (SITS) classification is a major research topic in remote sensing and is relevant for a wide range of applications. Deep learning approaches have been commonly employed for the SITS classification and have provided state-of-the-art performance. However, deep learning methods suffer from overfitting when labeled data are scarce. To address this problem, we propose a novel self-supervised pretraining scheme to initialize a transformer-based network by utilizing large-scale unlabeled data. In detail, the model is asked to predict randomly contaminated observations given an entire time series of a pixel. The main idea of our proposal is to leverage the inherent temporal structure of satellite time series to learn general-purpose spectral-temporal representations related to land cover semantics. Once pretraining is completed, the pretrained network can be further adapted to various SITS classification tasks by fine-tuning all the model parameters on small-scale task-related labeled data. In this way, the general knowledge and representations about SITS can be transferred to a label-scarce task, thereby improving the generalization performance of the model as well as reducing the risk of overfitting. Comprehensive experiments have been carried out on three benchmark datasets over large study areas. Experimental results demonstrate the effectiveness of the proposed pretraining scheme, leading to substantial improvements in classification accuracy using transformer, 1-D convolutional neural network, and bidirectional long short-term memory network. The code and the pretrained model will be available at https://github.com/linlei1214/SITS-BERT upon publication.

Subfields: Ecology, Media Technology, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Ecology

In Media Technology

In Signal Processing

Citations in the corpus

Continual Barlow Twins: Continual Self-Supervised Learning for Remote Sensing Semantic Segmentation (2022, background): “In [26], the authors show the effectiveness of a SSL pretraining for time-series classification.”
ClimaX: A foundation model for weather and climate (2023, unknown): “A few recent works have proposed pretraining techniques for satellite imagery and remote sensing (Yuan & Lin, 2020; Cong et al., 2022; Reed et al., 2022a) but they have so far not been applied to multi-sensory data and variables in weather and climate.”
Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “There have been many recent advancements for time-series Transformers for classiﬁcation tasks [65, 63, 62, 60, 68].”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, methodology): “Yuan and Lin [106] applied this idea to time-series Sentinel-2 images to learn spectral–temporal features related to crop semantics, as the phenological change pattern of crops on time-series RSIs can reflect the crop type.”
Self-Supervised Learning in Remote Sensing: A review (2022, methodology): “In practice, three common procedures are used for quantitative evaluation: • Linear probing (or linear classiﬁcation) : refers to ﬁxing the parameters of the learned model f (frozen encoder), and training a linear classiﬁer g on top of the generated representations.”
Remote Sensing Image Change Detection With Transformers (2021, background): “The astounding performance of the transformer models on NLP/CV tasks has intrigued the RS community to study their applications in RS tasks, such as image time-series classification [45], [46], hyperspectral image classification [47], scene classification [48], and RS image captioning [49], [50].”