Context dependent recurrent neural network language model

By Tomas Mikolov, Geoffry Zweig. 2012

624 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Exploring the Limits of Language Modeling (2016, neutral): “…to questions about object colors, facts about people, etc. Lastly, recently proposed sequence-to-sequence models employ conditional language models (Mikolov & Zweig, 2012) as their key component to solve diverse tasks like machine translation (Sutskever et al., 2014; Cho et al., 2014;…”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “We apply Transformer-XL to a variety of datasets on both word-level and character-level language modeling to have a comparison with state-of-the-art systems, including WikiText-103 (Merity et al., 2016), enwiki8 (LLC, 2009), text8 (LLC, 2009), One Billion Word (Chelba et al., 2013), and Penn Treeban”
A Convolutional Neural Network for Modelling Sentences (2014, methodology): “Besides comprising powerful classifiers as part of their architecture, neural sentence models can be used to condition a neural language model to generate sentences word by word (Schwenk, 2012; Mikolov and Zweig, 2012; Kalchbrenner and Blunsom, 2013a).”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, background): “However, it is suggested by the following observations: • Natural language is highly context-dependent (Mikolov & Zweig, 2012).”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, background): “The idea of capturing document-level semantic information has been previously explored for recurrent neural networks (Mikolov and Zweig, 2012; Ghosh et al., 2016; Dieng et al., 2017), however, we are not aware of any existing convolutional models.”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, methodology): “bility distribution of the next target using neural models (Mikolov and Zweig, 2012; Melis et al., 2017; Bengio et al., 2003) with:”
Language Models as Knowledge Bases? (2019, methodology): “…a probability p ( w ) to the sequence by factorizing it as follows A common way to estimate this probability is us-ing neural language models (Mikolov and Zweig, 2012; Melis et al., 2017; Bengio et al., 2003) with where h t ∈ R k is the output vector of a neural network at position t and W ∈…”
Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “Similarly, topic features have been used with either maximum entropy models [33] or recurrent networks [46, 53].”
Improving Neural Language Models with a Continuous Cache (2016, methodology): “RNN+LSA+KN5+cache (Mikolov & Zweig, 2012) 90.3 LSTM (Zaremba et al., 2014) 78.4 Variational LSTM (Gal & Ghahramani, 2015) 73.4 Recurrent Highway Network (Zilly et al., 2016) 66.0 Pointer Sentinel LSTM (Merity et al., 2016) 70.9 Training procedure.”
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021, background): “But longer sequences are more expensive to train on.2 Before transformers, RNN language models were trained on shorter-L sequences and assumed to generalize to longer contexts at inference time (Mikolov et al., 2010; Mikolov & Zweig, 2012; Zaremba et al., 2014).”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “We followed the same setting as in (Mikolov and Zweig, 2012) which resulted in 18.55K words for training set, 14.5K and 16K words in the validation and test sets respectively.”
Recurrent Continuous Translation Models (2013, background): “Word representations have also shown a marked sensitivity to conditioning information (Mikolov and Zweig, 2012).”
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (2015, background): “Mikolov & Zweig (2012) previously observed performance boosts for recurrent language models by adding the capacity to jointly learn a document-level representation.”
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015, methodology): “Recurrent neural networks have been successfully used in language modeling (Mikolov & Zweig, 2012), speech recognition, and sentence generation from images (Karpathy & Fei-Fei, 2015).”
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses (2015, methodology): “For example, Mikolov and Zweig (2012) and Auli et al. (2013) use a pre-trained topic model.”