Learning Precise Timing with LSTM Recurrent Networks

By Felix Alexander Gers, N. Schraudolph, J. Urgen Schmidhuber. 2003

1,717 citations recorded by the source corpus.

The temporal distance between events conveys information essential for numerous sequential tasks such as motor control and rhythm detection. While Hidden Markov Models tend to ignore this information, recurrent neural networks (RNNs) can in principle learn to make use of it. We focus on Long Short-Term Memory (LSTM) because it has been shown to outperform other RNNs on tasks involving long time lags. We find that LSTM augmented by "peephole connections" from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes spaced either 50 or 49 time steps apart without the help of any short training exemplars. Without external resets or teacher forcing, our LSTM variant also learns to generate stable streams of precisely timed spikes and other highly nonlinear periodic patterns. This makes LSTM a promising approach for tasks that require the accurate measurement or generation of time intervals.

Subfields: General Health Professions, Model Architectures, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In General Health Professions

In Model Architectures

In Philosophy

Citations in the corpus

Generating Sequences With Recurrent Neural Networks (2013, methodology): “For the version of LSTM used in this paper [7] H is implemented by the following composite function: it = σ (Wxixt +Whiht−1 +Wcict−1 + bi) (7) ft = σ (Wxfxt +Whfht−1 +Wcfct−1 + bf ) (8) ct = ftct−1 + it tanh (Wxcxt +Whcht−1 + bc) (9) ot = σ (Wxoxt +Whoht−1 +Wcoct + bo) (10) ht = ot tanh(ct)…”
Speech recognition with deep recurrent neural networks (2013, methodology): “For the version of LSTM used in this paper [14] H is implemented by the following composite function:”
Towards Universal Paraphrastic Sentence Embeddings (2015, methodology): “We use the version from Gers et al. (2003) which has the following equations: where σ is the logistic sigmoid function.”
Beyond short snippets: Deep networks for video classification (2015, background): “Unlike standard RNNs, the Long Short Term Memory (LSTM) architecture [6] uses memory cells (Figure 3) to store and output information, allowing it to better discover long-range temporal relationships.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, neutral): “…techniques for recurrent networks have been introduced and continue to be actively explored (El Hihi & Bengio, 1995; Schuster & Paliwal, 1997; Gers et al., 2002; Koutnik et al., 2014; Le et al., 2015; Ba et al., 2016; Wu et al., 2016; Krueger et al., 2017; Merity et al., 2017; Campos et al.,…”
A Deep Learning Approach for Optimization of Systematic Signal Detection in Financial Trading Systems with Big Data (2017, methodology): “As a type of recurrent neural network (RNN), LSTM [31], [32] is a powerful method for analysing time series data with gaps of unknown size between important events.”
Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks (2017, background): “storing and accessing information over very long timespans [13].”
DOC: Deep Open Classification of Text Documents (2017, background): “Note: other existing deep models like RNN (Williams and Zipser, 1989; Schus-ter and Paliwal, 1997) and LSTM (Hochreiter and Schmidhuber, 1997; Gers et al., 2002) can also be adopted as the base.”
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF (2016, background): “It should be noted that we do not include peephole connections (Gers et al., 2003) in the our LSTM formulation.”
Learning to Diagnose with LSTM Recurrent Neural Networks (2015, methodology): “Our proposed LSTM RNN uses memory cells with forget gates (Gers et al., 2000) but without peephole connections (Gers et al., 2003).”
EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding (2015, neutral): “We omit the→ arrow for uncluttered formulation. it = σ(Wixxt + Wihht−1 + Wicct−1 + bi) (3a) ft = σ(Wfxxt + Wfhht−1 + Wfcct−1 + bf ) (3b) ct = ft ct−1 + it φ(Wcxxt + Wchht−1 + bc) (3c) ot = σ(Woxxt + Wohht−1 + Wocct + bo) (3d) ht = ot φ(ct) (3e) where it, ot, ft, ct are the activation of the input…”
Learning Simple Algorithms from Examples (2015, methodology): “However, for the LSTM controllers, this structure is only loosely present in the matrix, indicating that they have failed to learn the correct algorithm. training according to: Taking the action a in state s causes a transition to state s (cid:48) , which in our case is deterministic.”
Charagram: Embedding Words and Sentences via Character n-grams (2016, methodology): “First we consider LSTM architectures (Hochreiter and Schmidhuber, 1997) over the character sequence x , using the version from Gers et al. (2003).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “2) will develop high-level, abstract, spatio-temporal feature detectors that may be active for thousands of time steps, as long as those memories are useful to predict (and thus compress) future observations [62, 61, 189, 79].”
Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings (2017, background): “Several others have pursued this goal (Socher et al., 2011; Le and Mikolov, 2014; Pham et al., 2015; Kiros et al., 2015; Hill et al., 2016; Arora et al., 2017; Pagliardini et al., 2017), though usually with the intent to extract useful features for supervised sentence tasks rather than to capture…”
Batch normalized recurrent neural networks (2015, methodology): “More concretely, we define the computation at time step t as follows [15]:”
Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval (2015, methodology): “One of the effective solutions for this problem in RNNs is using memory cells instead of neurons originally proposed in [15] as Long Short-Term Memory (LSTM) and completed in [30] and [31] by adding forget gate and peephole connections to the architecture.”