Generating Sequences With Recurrent Neural Networks

By Alex Graves. 2013

4,326 citations recorded by the source corpus.

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

Subfields: Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Attention is All you Need (2017, methodology): “At each step the model is auto-regressive [9], consuming the previously generated symbols as additional input when generating the next.”
Adam: A Method for Stochastic Optimization (2014, methodology): “Optimization methods bearing a direct relation to Adam are RMSProp (Tieleman & Hinton, 2012; Graves, 2013) and AdaGrad (Duchi et al., 2011); these relationships are discussed below.”
Neural Machine Translation in Linear Time (2016, positive): “All the results except for the ByteNet result are obtained using some variant of the LSTM recurrent neural network (Hochreiter & Schmidhuber, 1997).”
Layer Normalization (2016, methodology): “We used the same model architecture as in Section (5.2) of Graves [2013].”
Exploring the Limits of Language Modeling (2016, methodology): “All experiments were run using the TensorFlow system (Abadi et al., 2015), with the exception of some older models which were used in the ensemble.”
Long Short-Term Memory-Networks for Machine Reading (2016, methodology): “It is possible to construct deep LSTMNs by stacking multiple memory and hidden layers in an alternating fashion, resembling a stacked LSTM (Graves, 2013) or a multi-hop memory network (Sukhbaatar et al.”
Grammar as a Foreign Language (2014, background): “[28] was the first to propose a differentiable attention mechanism for the general problem of handwritten text synthesis, although his approach assumed a monotonic alignment between the input and output sequences.”
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014, methodology): “We follow the implementation of LSTM as used in Graves [2013].”
Sequence to Sequence Learning with Neural Networks (2014, methodology): “Graves [10] introduced a novel differentiable attention mechanism that allows neural networks to focus on different parts of their input, and an elegant variant of this idea was successfully applied to machine translation by Bahdanau et al. [2].”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, background): “A similar approach of aligning an output symbol with an input symbol was proposed recently by Graves (2013) in the context of handwriting synthesis.”
Semi-supervised Sequence Learning (2015, methodology): “In our experiments with LSTMs, we follow the basic recipes as described in [7, 32] by clipping the cell outputs and gradients.”
Training Very Deep Networks (2015, background): “Skip connections between layers or to output layers (where error is “injected”) have long been used in neural networks, more recently with the explicit aim to improve the ﬂow of information [20, 21, 2, 22].”
Deep AutoRegressive Networks (2013, methodology): “Training was done with RMSprop (Graves, 2013), momentum 0.9 and minibatches of size 100.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Self-attention is a variant of attention [Graves, 2013; Bahdanau et al., 2015] that processes a sequence by replacing each element by a weighted average of the rest of the sequence.”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We take a PUE of 1.1 and a carbon intensity factor set at the national US average of 0.385 kg CO 2 e per KWh. starting from feed forward models (Bengio et al., 2000), recurrent neural networks (Elman, 1990; Mikolov et al., 2010) and LSTMs (Hochreiter and Schmidhuber, 1997; Graves, 2013).”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, methodology): “Once a language model is learned, the model can be used to generate sequences that capture long-range dependencies (Graves, 2013).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “, 2001), and the introduction of gating in LSTMs and the gradient clipping technique (Graves, 2013) might not be sufficient to fully address this issue.”
Universal Transformers (2018, methodology): “Like the Transformer model, the UT is autoregressive (Graves, 2013).”
Adaptive Computation Time for Recurrent Neural Networks (2016, methodology): “Following previous RNN experiments on the same data [8], the raw unicode text was used, including XML tags and markup characters, with one byte presented per input timestep and the next byte predicted as a target.”