Recurrent neural network based language model

By Tomas Mikolov, Martin Karafiát, Lukáš Burget, Jan Honza Cernocký, Sanjeev Khudanpur. 2010

6,125 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Neural Machine Translation in Linear Time (2016, background): “Recurrent neural networks (RNN) are powerful sequence models (Hochreiter & Schmidhuber, 1997) and are widely used in language modelling (Mikolov et al., 2010), yet they have a potential drawback.”
Exploring the Limits of Language Modeling (2016, background): “Much work has been done on small and large scale RNN-based LMs (Mikolov et al., 2010; Mikolov, 2012; Chelba et al., 2013; Zaremba et al., 2014; Williams et al., 2015; Ji et al., 2015a; Wang & Cho, 2015; Ji et al., 2015b).”
Long Short-Term Memory-Networks for Machine Reading (2016, background): “…or vanishing gradients (Bengio et al., 1994), leading to models with gated activation functions (Hochreiter and Schmidhuber, 1997; Cho et al., 2014), and more advanced architectures that enhance the information flow within the network (Koutnı́k et al., 2014; Chung et al., 2015; Yao et al., 2015).”
Sequence to Sequence Learning with Neural Networks (2014, methodology): “So far, the simplest and most effective way of applying an RNN-Language Model (RNNLM) [23] or a Feedforward Neural Network Language Model (NNLM) [3] to an MT task is by rescoring the nbest lists of a strong MT baseline [22], which reliably improves translation quality.”
Character-Level Language Modeling with Deeper Self-Attention (2018, background): “By far, the most popular architecture in this area is the RNN and variants, ﬁrst studied in [32].”
MaskGAN: Better Text Generation via Filling in the ______ (2018, background): “They have shown impressive results in language modeling (Mikolov et al., 2010), machine translation (Wu et al., 2016) and text classiﬁcation (Miyato et al., 2017).”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “These encoders are typically implemented as convolutional (Kim, 2014), recursive (Socher et al., 2013), or recurrent neural networks (Mikolov et al., 2010) operating over a sentence’s words or characters (Zhang et al., 2015; Kim et al., 2016).”
Semi-supervised Sequence Learning (2015, methodology): “We also ﬁnd that recurrent language models [23] can be used as a pretraining method for LSTMs.”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “, 2000), recurrent neural networks (Elman, 1990; Mikolov et al., 2010) and LSTMs (Hochreiter and Schmidhuber, 1997; Graves, 2013).”
A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “1 DATASETS We evaluate on several datasets commonly used for (word-level) language modeling: Penn Treebank (Mikolov et al., 2010), WikiText-2 (Bradbury et al.”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “Actually, the masked language modeling in BERT (Devlin et al., 2018) and the standard language modeling (Bengio et al., 2003; Mikolov et al., 2010) in GPT (Radford et al., 2018) can be viewed as special cases of MASS.”
Cross-lingual Language Model Pretraining (2019, background): “While recurrent neural networks obtain state-of-the-art performance on language modeling benchmarks (Mikolov et al., 2010; Jozefowicz et al., 2016), Transformer models are also very competitive (Dai et al., 2019).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “However, different from truncated BPTT, our method caches a sequence of hidden states instead of the last one, and should be applied together with the relative positional encoding technique described in Section 3.3.”
Efficient Estimation of Word Representations in Vector Space (2013, background): “Recurrent neural network based language model has been proposed to overcome certain limitations of the feedforward NNLM, such as the need to specify the context length (the order of the model N ), and because theoretically RNNs can efficiently represent more complex patterns than the shallow neural ”
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, background): “Schmaltz showed that recurrent neural network language models [11] with long short-term memory [12] cells work effectively for word ordering even without any explicit syntactic information.”