A Critical Review of Recurrent Neural Networks for Sequence Learning

By Zachary Chase Lipton. 2015

2,647 citations recorded by the source corpus.

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feedforward neural networks, recurrent networks retain a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have traditionally been dicult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. In this survey, we review and synthesize the research that over the past three decades rst yielded and then made practical these powerful learning models. When appropriate, we reconcile conicting notation and nomenclature. Our goal is to provide a selfcontained explication of the state of the art together with a historical perspective and references to primary research.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Interpretable and Steerable Sequence Learning via Prototypes (2019, background): “Recently, variants of RNNs including Long-Short Term Memory networks (LSTMs) [21] have been proven to be very effective in modeling sequence data.”
Learning to Diagnose with LSTM Recurrent Neural Networks (2015, background): “Surveys of the literature include Graves (2012), a thorough dissertation on sequence labeling with RNNs, De Mulder et al. (2015), which surveys natural language applications, and Lipton et al. (2015), which provides a broad overview of RNNs for sequence learning, focusing on modern applications.”
Deep learning for computational biology (2016, positive): “These include convolutional neural networks, which are widely used for modelling images (Box 2), recurrent neural networks for sequential data (Sutskever, 2013; Lipton, 2015), or restricted Boltzmann machines (Salakhutdinov & Larochelle, 2010; Hinton, 2012) and autoencoders (Hinton & Salakhutdinov, ”
Low-Cost Recurrent Neural Network Expected Performance Evaluation (2018, background): “Input at time t − 1 (xt−1) can determine the output at time t (yt) by way of recurrent connections [20].”
Deep Air Quality Forecasting Using Hybrid Deep Learning Framework (2018, background): “More recently, deep learning has been widely applied to sequence data processing and time series problems [20], [21], [24].”
An Introductory Review of Deep Learning for Prediction Models With Big Data (2020, background): “The CEC is regulated by a forget gate and it can also be reset by the forget gate.”
Parallel Recurrent Neural Network Architectures for Feature-rich Session-based Recommendations (2016, methodology): “RNNs have been shown to perform excellent in modeling sequence data [15].”
Transformers in Time-Series Analysis: A Tutorial (2022, background): “The most notable of these machine learning models are Recurrent Neural Networks (RNNs) and their popular variants, including Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) [9], [10], [11], [12].”
Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks (2017, background): “RNNs are the deep models of choice when dealing with sequential data [12].”
Intelligent nanophotonics: merging photonics and artificial intelligence at the nanoscale (2018, background): “ng data or modifying the network structure. Recurrent neural networks (RNNs) that can learn from sequential inputs have been realized on a photonic platform very recently but not yet in nanophotonics [181,182]. Including time domain features in the dataset will be very attractive. Furthermore, other”
Knowledge Tracing: A Survey (2022, methodology): “It employs Recurrent Neural Network (RNN) [63] and a Long Short Term Memory ACM Computing Surveys, Vol. 55, No. 11, Article 224.”
Sequence-Aware Recommender Systems (2018, background): “10See [74] for an overview of Recurrent Neural Networks.”
Preparing Medical Imaging Data for Machine Learning (2020, background): “One solution is to extract information from the report of imaging findings through rulebased natural language processing (50,51) or recurrent neural networks (52,53).”
ChatGPT: Jack of all trades, master of none (2023, background): “In addition, their architecture did not allow for efficient parallelization of training and inference processes [8].”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, background): “The long short-term memory (LSTM) networks alleviate this shortcoming with special hidden units known as “gates” that can effectively control the scale of information to remember or forget in the backpropagation [51].”
Data-driven discovery of coordinates and governing equations (2019, background): “Other approaches use a recurrent NN architecture, particularly long short-term memory (LSTM) networks, for applications involving sequential data [68–70].”