Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics

By Pantelis R. Vlachas, Jaideep Pathak, Brian R. Hunt, Themistoklis P. Sapsis, Michelle Girvan, Edward Ott, Petros Koumoutsakos. Neural Networks, 2019

503 citations recorded by the source corpus.

We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the long-term forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto-Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/f78065c75a1d0448fc0fee1dfef8cb93c607c1cd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Daniel Brunner
2. Aaron Griffith
3. Sergei K. Turitsyn
4. Daniel J. Gauthier
5. Erik M. Bollt
6. Wendson A. S. Barbosa

In Statistical and Nonlinear Physics

1. Sergei K. Turitsyn
2. John M. Dudley
3. Goëry Genty
4. Erik M. Bollt
5. Santo Fortunato
6. Jakob Runge

Citations in the corpus

• Emerging opportunities and challenges for the future of reservoir computing (2024, methodology): “Furthermore, some-timesthosedesignsarethemselvescoupledwiththewaytheinputand output parts of the system are used, for example in solving partial differential equations (PDEs) 21,22 or representing the dynamics of multivariate time series 23 .”
• Machine learning and applications in ultrafast photonics (2020, background): “As research continues to progress both in the development of machine-learning algorithms and ultrafast photonics technologies, we can expect even more fruitful interactions with increased influence of the former in the physical understanding, design, optimization and operation of the latter.”
• Next generation reservoir computing (2021, methodology): “(7), where the vector field is the right-hand-side of the differential equations.”