Statistical Language Models Based on Neural Networks

By Vysoké Učení, Technické V Brně, Grafiky A Multimédií, Disertační Práce. 2012

660 citations recorded by the source corpus.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, methodology): “Recurrent Neural Networks based LMs employ the chain rule to model joint probabilities over word sequences: where the context of all previous words is encoded with an LSTM, and the probability over words uses a Softmax (see Figure 1(a)).”
Sequence to Sequence Learning with Neural Networks (2014, methodology): “Feedforward Neural Network Language Model (NNLM) [3] to an MT task is by rescoring the nbest lists of a strong MT baseline [22], which reliably improves translation quality.”
Generating Sequences With Recurrent Neural Networks (2013, methodology): “For example, he records a perplexity of 141 for a 5-gram with KeyserNey smoothing, 141.8 for a word level feedforward neural network, 131.1 for the state-of-the-art compression algorithm PAQ8 and 123.2 for a dynamically evaluated word-level RNN.”
Distributed Representations of Sentences and Documents (2014, background): “Distributed representations for words were ﬁrst proposed in (Rumelhart et al., 1986) and have become a successful paradigm, especially for statistical language modeling (El-man, 1990; Bengio et al., 2006; Mikolov, 2012).”
One billion word benchmark for measuring progress in statistical language modeling (2013, background): “The Recurrent Neural Network (RNN) based LM have recently achieved outstanding performance on a number of tasks (Mikolov, 2012).”
Distributed Representations of Words and Phrases and their Compositionality (2013, background): “The follow up work includes applications to automatic speech recognition and machine translation [14, 7], and a wide range of NLP tasks [2, 20, 15, 3, 18, 19, 9].”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 26, 21, 18, 9], and some of the resulting word vectors were made available for future research and comparison2.”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “While this has made them interesting for language modeling (Mikolov, 2012), it may not have the capacity to learn how the algorithmic patterns are generated.”
Learning Visual N-Grams from Web Data (2016, background): “Visual n-gram models are less suitable for caption generation5 [44] but they are much more efficient to evaluate at inference time, which is very important in real-world applications of these models.”
Learning to Execute (2014, methodology): “From a methodological perspective, we formulate the program evaluation task as a language modeling problem on a sequence (Mikolov, 2012; Sutskever, 2013; Pascanu et al., 2013).”
FastText.zip: Compressing text classification models (2016, background): “One of the main successes of deep learning is due to the effectiveness of recurrent networks for language modeling and their application to speech recognition and machine translation (Mikolov, 2012).”
On the difficulty of training recurrent neural networks (2012, methodology): “Therefore studying the asymptotic behaviour of ˜ F can provide useful information about where such events are likely to happen.”
A Neural Conversational Model (2015, background): “Our work is also inspired by the recent success of neural language modeling (Bengio et al., 2003; Mikolov et al., 2010; Mikolov, 2012), which shows that recurrent neural networks are rather effective models for natural language.”
Recurrent Neural Network Regularization (2014, neutral): “The Recurrent Neural Network (RNN) is neural sequence model that achieves state of the art performance on important tasks that include language modeling (Mikolov, 2012), speech recognition (Graves et al., 2013), and machine translation (Kalchbrenner & Blunsom, 2013).”
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015, background): “At each time step, the hidden state of the Bidirectional LSTM is the concatenation of the forward and backward hidden states.”
How to Construct Deep Recurrent Neural Networks (2013, methodology): “RNNs have been successfully used for various task such as language modeling (see, e.g., Graves, 2013; Pascanu et al. , 2013a; Mikolov, 2012; Sutskever et al. , 2011), learning word embeddings (see, e.g., Mikolov et al. , 2013a), online handwritten recognition (Graves et al. , 2009) and speech…”
Exploiting Similarities among Languages for Machine Translation (2013, background): “It is unlike traditional neural network based language models (Bengio et al., 2003; Mnih and Hinton, 2008; Mikolov et al., 2010), where the objective is to predict the next word given the context of several preceding words.”
Visualizing and Understanding Recurrent Networks (2015, methodology): “This observation is related to the improvements seen in “dynamic evaluation” (21; 17) of recurrent language models, where an RNN is allowed to train on the test set characters during evaluation as long as it sees them only once.”
Colorless Green Recurrent Networks Dream Hierarchically (2018, methodology): “As a proof of concept, Pereira (2000) and, later, Mikolov (2012) computed the probability of Chomsky’s famous “colorless green ideas” sentence using a class-based bigram LM and an RNN, respectively, and showed that it is much higher than the probability of its shufﬂed ungrammatical variants.”
Gated Feedback Recurrent Neural Networks (2015, background): “Both of these units are central to our proposed model, and we will describe them in more details in the remainder of this section.”