Generating Text with Recurrent Neural Networks

By Ilya Sutskever, James Martens, Geoffrey E. Hinton. 2011

1,560 citations recorded by the source corpus.

Subfields: Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Generative Models

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “A class of models that solve this problem more efﬁciently are character-level LSTMs (Sutskever et al., 2011; Graves, 2013).”
Generating Sequences With Recurrent Neural Networks (2013, background): “Character-level language modelling with neural networks has recently been considered [30, 24], and found to give slightly worse performance than equivalent word-level models.”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “, 2015), and for large px the objective is similar to word-level ‘debagging’ (Sutskever et al., 2011).”
Revisiting Natural Gradient for Deep Networks (2013, methodology): “Hessian-Free optimization (Martens, 2010; Sutskever et al. , 2011; Chapelle and Erhan, 2011), Krylov Sub-space Descent (Vinyals and Povey, 2012), natural gradient descent (Amari, 1997; Park et al. , 2000), TONGA (Le Roux et al. , 2008; Roux and Fitzgibbon, 2010) are just a few of such recently…”
Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function (2019, methodology): “This definition was first used by Zhao et al. (2017) and later adapted by Bordia and Bowman (2019).”
Pixel Recurrent Neural Networks (2016, methodology): “RNNs have been shown to excel at hard sequence problems ranging from handwriting generation (Graves, 2013), to character prediction (Sutskever et al., 2011) and to machine translation (Kalchbrenner & Blunsom, 2013).”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “Language models: Neural language models (Mikolov et al., 2010; Sutskever et al., 2011) have been shown to be useful for many natural language processing tasks.”
Program Synthesis with Large Language Models (2021, methodology): “Our work is inspired by the long line of previous work on neural language models of natural language text [Mikolov et al., 2010, Sutskever et al., 2011, Józefowicz et al., 2016, Dai and Le, 2015, Peters et al., 2018, Howard and Ruder, 2018], especially recent large Transformer models [Radford et…”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “While early work used feed-forward networks with a fixed-length history window, Mikolov et al. (2010); Sutskever et al. (2011); Graves (2013) proposed to use recurrent neural networks instead and found that this significantly improved performance.”
Enriching Word Vectors with Subword Information (2016, background): “A first class of such models are recurrent neural networks, applied to language modeling (Mikolov et al., 2012; Sutskever et al., 2011; Graves, 2013; Bojanowski et al., 2015), text normalization (Chrupała, 2014), part-ofspeech tagging (Ling et al.”
Neural Turing Machines (2014, background): “, 2013; Graves and Jaitly, 2014), text generation (Sutskever et al., 2011), handwriting generation (Graves, 2013) and machine translation (Sutskever et al.”
Autoregressive Entity Retrieval (2020, methodology): “, maximizing the output sequence likelihood with teacher forcing (Sutskever et al., 2011; 2014) and regularized with dropout (Srivastava et al.”
HyperNetworks (2016, methodology): “…2016) 1.32 Recurrent Dropout LSTM (Semeniuta et al., 2016) 1.301 1.338 Zoneout RNN (Krueger et al., 2016) 1.27 HM-LSTM 3 (Chung et al., 2016) 1.27 (Sutskever et al., 2011) 1.60 GF-RNN (Chung et al., 2015) 1.58 20.0 M Grid-LSTM (Kalchbrenner et al., 2016) 1.47 16.8 M LSTM (Rocki, 2016b) 1.45…”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
On the difficulty of training recurrent neural networks (2012, methodology): “Sutskever et al. (2011) use the Hessian-Free optimizer in conjunction with structural damping , a spe-ciﬁc damping strategy of the Hessian.”
Long-term recurrent convolutional networks for visual recognition and description (2014, background): “Though RNNs have proven successful on tasks such as speech recognition [43] and text generation [38], it can be difficult to train them to learn long-term dynamics, likely due in part to the vanishing and exploding gradients problem [12] that can result from propagating the gradients down through th”
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (2016, methodology): “Commonly used approaches is to either compare the performance of the sentence embeddings on down-stream tasks (Hill et al., 2016), or to analyze models, speciﬁcally trained for predeﬁned task (Schmaltz et al., 2016; Sutskever et al., 2011).”