Extensions of recurrent neural network language model

By Tomas Mikolov, Stefan Kombrink, Lukáš Burget, Jan Honza Cernocký, Sanjeev Khudanpur. 2011

1,624 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, methodology): “…is different when training the small NNLM model on either the Penn Treebank (Marcus et al., 1993) or text8 datasets (for PTB, we used the same train/validation/test set split and vocabulary as (Mikolov et al., 2011), while on text8 we used the split/vocabulary from (Mikolov et al., 2014)).”
Distributed Representations of Words and Phrases and their Compositionality (2013, methodology): “It has been observed before that grouping words together by their f requency works well as a very simple speedup technique for the neural network based language mod ls [5, 8].”
Generating Sentences from a Continuous Space (2015, background): “Recurrent neural network language models (rnnlms, Mikolov et al., 2011) represent the state of the art in unsupervised generative modeling for natural language sentences.”
A Convolutional Neural Network for Modelling Sentences (2014, background): “The Recurrent Neural Network (RNN) is a special case of the recursive network where the structure that is followed is a simple linear chain (Gers and Schmidhuber, 2001; Mikolov et al., 2011).”
Adaptive Input Representations for Neural Language Modeling (2018, background): “The hierarchical softmax does this by introducing latent variables or clusters to simplify normalization (Goodman, 2001; Morin & Bengio, 2005; Mikolov et al., 2011).”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Several self-supervision signals have been proposed, such as autoregressive language modeling [5, 52], which includes the GPT series of models [6, 58, 59], and masked language modeling, which includes ELMo [56] and BERT-related methods [16,32,47].”
Recurrent Continuous Translation Models (2013, methodology): “To speed up training, we factorize the target vocabulary V F into 256 classes following the procedure in (Mikolov et al., 2011).”
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems (2015, methodology): “An l 2 regularisation term was added to the objective function for every 10 training examples as suggested in Mikolov et al. (2011b).”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “The PPL , B-1, B-2, B-3 and B-4 of the generated sentences using our m-RNN-AlexNet model in this dataset are 24.39, 0.565, 0.386, 0.256, and 0.170 respectively.”
Efficient softmax approximation for GPUs (2016, methodology): “Similar to the frequency binning hierarchical softmax (Mikolov et al., 2011c), this information can be exploited to reduce the computation cost.”
How to Construct Deep Recurrent Neural Networks (2013, methodology): “…and the sRNN for word-level modeling surpassed the previous best performance achieved by an RNN with 1000 long short-term memory (LSTM) units (Graves, 2013) as well as that by a shallow RNN with a larger hidden state (Mikolov et al. , 2011), even when both of them used dynamic evaluation 3 .”
Explain Images with Multimodal Recurrent Neural Networks (2014, methodology): “We briefly introduce the simple Recurrent Neural Network (RNN) or Elman network [5] that is widely used for many natural language processing tasks, such as speech recognition [25, 26].”
Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems (2015, background): “Recurrent Neural Networks (RNNs) have proven successful at several tasks involving natural language, language modeling (Mikolov et al., 2011), and have been applied recently to dialog (Sordoni et al., 2015; Vinyals & Le, 2015; Shang et al., 2015).”
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, background): “Ostensibly, this appears to be true: Peters et al. (2018) show that using pretrained deep word representations achieves state of the art on a variety of tasks.”
Learning Longer Memory in Recurrent Neural Networks (2014, positive): “The pre-processing of data and division to training, validation and test parts are the same as in (Mikolov et al., 2011).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, methodology): “This template is naturally suited for many NLP tasks such as language modeling [2, 64, 65], machine translation [66, 67, 68], speech recognition [69, 70, 71, 72], image captioning [73].”