Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery

By Lichao Mou, Lorenzo Bruzzone, Xiaoxiang Zhu. IEEE Transactions on Geoscience and Remote Sensing, 2018

559 citations recorded by the source corpus.

Change detection is one of the central problems in earth observation and was extensively investigated over recent decades. In this paper, we propose a novel recurrent convolutional neural network (ReCNN) architecture, which is trained to learn a joint spectral–spatial–temporal feature representation in a unified framework for change detection in multispectral images. To this end, we bring together a convolutional neural network and a recurrent neural network into one end-to-end network. The former is able to generate rich spectral-spatial feature representations, while the latter effectively analyzes temporal dependence in bitemporal images. In comparison with previous approaches to change detection, the proposed network architecture possesses three distinctive properties: 1) it is end-to-end trainable, in contrast to most existing methods whose components are separately trained or computed; 2) it naturally harnesses spatial information that has been proven to be beneficial to change detection task; and 3) it is capable of adaptively learning the temporal dependence between multitemporal images, unlike most of the algorithms that use fairly simple operation like image differencing or stacking. As far as we know, this is the first time that a recurrent convolutional network architecture has been proposed for multitemporal remote sensing image analysis. The proposed network is validated on real multispectral data sets. Both visual and quantitative analyses of the experimental results demonstrate competitive performance in the proposed mode.

Subfields: Analytical Chemistry, Atmospheric Science, Media Technology

View Paper: https://www.semanticscholar.org/paper/4ec33bbb1940aaa52dccac225b02f5533ba7aec1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Atmospheric Science

In Media Technology

Citations in the corpus

Deep learning for change detection in remote sensing: a review (2022, methodology): “Recently, one exception is the work of Mou and Zhu (2018) who managed to extract joint spatial-temporal features for land use change detection in complex urban areas using Landsat images by combining CNN and RNN.”
Self-Attention for Raw Optical Satellite Time Series Classification (2019, methodology): “and land cover [25], as well as for crop type identiﬁcation [54; 59; 64; 65]. To utilize both spatial and temporal features from time series, combinations of convolutional layers and recurrent layers [58; 57] or convolutional-recurrent networks [66] have been explored and comprehensively compared [6”
A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection (2020, methodology): “The spatial–temporal relationships have been discussed and proven effective in some recent studies [29,31], where RNN is used to model such relationships.”
Optical Remote Sensing Image Change Detection Based on Attention Mechanism and Image Difference (2021, background): “Many scholars have introduced deep learning technology to remote sensing processing and have achieved excellent performance in image classification [25], [26], semantic segmentation [27]–[29], object detection [30], [31], change detection [17], [32]–[35], and other tasks.”
DynamicEarthNet: Daily Multi-Spectral Satellite Dataset for Semantic Change Segmentation (2022, background): “Among these works, [25, 28] directly predict the multi-class change map whereas, [36] define change as the difference between two semantic maps.”
Self-Supervised Representation Learning for Remote Sensing Image Change Detection Based on Temporal Prediction (2020, methodology): “Following them, deep learning-based CD methods have been widely explored [48].”
SemiCDNet: A Semisupervised Convolutional Neural Network for Change Detection in High Resolution Remote-Sensing Images (2020, unknown): “unknown pixels can be determined directly [32].”
End-to-End Change Detection for High Resolution Satellite Images Using Improved UNet++ (2019, methodology): “Mou et al. [49] further proposed an end-to-end CD network by combining the CNN with the RNN, where a joint spectral-spatial-temporal feature representation is learned.”
Multitask learning for large-scale semantic change detection (2018, methodology): “Mou et al. (2019) proposed using patch based recurrent CNNs to detect changes in image pairs.”
Change Detection in Multisource VHR Images via Deep Siamese Convolutional Multiple-Layers Recurrent Neural Network (2020, background): “[28], the DSCNN part of SiamCRNN has a ﬂexible deep siamese structure for multisource data, and the MRNN in SiamCRNN aims to fully mine the change information between heterogeneous images.”
A Deeply Supervised Attention Metric-Based Network and an Open Aerial Image Dataset for Remote Sensing Change Detection (2021, background): “[46], which is able to learn better feature representations with joint spectral-spatial-temporal information from multispectral images for CD using a very small input size of 5 × 5.”
Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series (2018, unknown): “These types of combinations have also been used for land cover change detection task between multi-spectral images [49,50].”