Self-Attention for Raw Optical Satellite Time Series Classification

By Marc Rußwurm, Marco Körner. arXiv.org, 2019

345 citations recorded by the source corpus.

The amount of available Earth observation data has increased dramatically in the recent years. Efficiently making use of the entire body information is a current challenge in remote sensing and demands for light-weight problem-agnostic models that do not require region- or problem-specific expert knowledge. End-to-end trained deep learning models can make use of raw sensory data by learning feature extraction and classification in one step solely from data. Still, many methods proposed in remote sensing research require implicit feature extraction through data preprocessing or explicit design of features. In this work, we compare recent deep learning models on crop type classification on raw and preprocessed Sentinel 2 data. We concentrate on the common neural network architectures for time series, i.e., 1D-convolutions, recurrence, a shallow random forest baseline, and focus on the novel self-attention architecture. Our central findings are that data preprocessing still increased the overall classification performance for all models while the choice of model was less crucial. Self-attention and recurrent neural networks, by their architecture, outperformed convolutional neural networks on raw satellite time series. We explore this by a feature importance analysis based on gradient back-propagation that exploits the differentiable nature of deep learning models. Further, we qualitatively show how self-attention scores focus selectively on few classification-relevant observations.

Subfields: Ecology, Media Technology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/1fcf78c4757412bfa0bb312dd4a6cbe9bc830c27

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Ecology

In Media Technology

In Signal Processing

Citations in the corpus

Transformers in Time-Series Analysis: A Tutorial (2022, methodology): “The classification of vegetation type [49] from raw optical satellite images is done using Transformers.”
SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery (2022, methodology): “Moreover, automatic analysis of satellite imagery is often needed for tasks with large societal impact such as poverty or crop yield prediction [21, 22, 23, 24, 25, 26, 27, 28, 29, 30], where acquiring large amounts of labeled data through surveys is impossible or prohibitively expensive.”
Self-Supervised Pretraining of Transformers for Satellite Image Time Series Classification (2020, methodology): “In this section, we validate its effectiveness on Bi-LSTM, which performed quite well in SITS classification [27].”