Improved backing-off for M-gram language modeling

By Reinhard Kneser, Hermann Ney. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, methodology): “Recurrent Neural Networks based LMs employ the chain rule to model joint probabilities over word sequences: where the context of all previous words is encoded with an LSTM, and the probability over words uses a Softmax (see Figure 1(a)).”
LLaMA: Open and Efficient Foundation Language Models (2023, background): “, 1983), and various smoothing techniques were proposed to improve the estimation of rare events (Katz, 1987; Kneser and Ney, 1995).”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “First, as considered by a lot of prior work, one can employ a non-parametric model, namely an Ngram model (Kneser & Ney, 1995).”
A Survey of Machine Learning for Big Code and Naturalness (2017, methodology): “The use of n-gram LMs in software engineering originated with the pioneering work of Hindle et al. [88], who used an n-gram LM with Kneser-Ney [104] smoothing.”
Unbounded cache model for online language modeling with open vocabulary (2017, background): “In particular, smoothed N-gram models [32, 34] have been the dominant type of models historically, achieving good performance in practice [44].”
Improving Neural Language Models with a Continuous Cache (2016, neutral): “In particular, smoothed N-gram models (Katz, 1987; Kneser & Ney, 1995) achieve good performance in practice (Mikolov et al., 2011).”
Learning Visual N-Grams from Web Data (2016, methodology): “Our models are convolutional networks trained using a loss function that is motivated by n-gram smoothers commonly used in language modeling [26, 34]: we develop a novel, differentiable loss function that optimizes trainable parameters for frequent n-grams, whereas for infrequent n-grams, the loss i”
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation (2019, methodology): “We report performance of hybrid systems using both a 4-gram count based language model [20] and an LSTM language model [21] in the first pass decoding [22].”
Librispeech: An ASR corpus based on public domain audio books (2015, methodology): “Modified Kneser-Ney smoothed 3- a nd 4-grams [21, 22] are trained.”
Language Modeling with Gated Convolutional Networks (2016, background): “Recently, neural networks (Bengio et al., 2003; Mikolov et al., 2010; Jozefowicz et al., 2016) outperform classical n-gram language models (Kneser & Ney, 1995; Chen & Goodman, 1996).”
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses (2015, methodology): “Our MT feature set includes the following features that are common in Moses: forward and backward maximum likelihood “translation” probabilities, word and phrase penalties, linear distortion, and a modified Kneser-Ney language model (Kneser and Ney, 1995) trained on Twitter responses.”
Efficient softmax approximation for GPUs (2016, methodology): “These approaches are more computationally intensive but generalize better than traditional non-parametric models (Bahl et al., 1983; Kneser & Ney, 1995).”
Fast and Robust Neural Network Joint Models for Statistical Machine Translation (2014, methodology): “It is clear that this model is effectively an ( n + m ) -gram LM, and a 15-gram LM would be far too sparse for standard probability models such as Kneser-Ney back-off (Kneser and Ney, 1995) or Maximum Entropy (Rosenfeld, 1996).”
Attention-over-Attention Neural Networks for Reading Comprehension (2016, methodology): “All language model features are trained on the training proportion of each dataset, with 8-gram wordbased setting and Kneser-Ney smoothing (Kneser and Ney, 1995) trained by SRILM toolkit (Stolcke, 2002).”
Colorless Green Recurrent Networks Dream Hierarchically (2018, methodology): “…form in the training corpus out of the two candidate target forms (singular or plural); second, a 5-gram model with Kneser-Ney smoothing ( KN , Kneser and Ney, 1995) trained using the IRSTLM package (Federico et al., 2008) and queried us-ing KenLM (Heaﬁeld, 2011); and third, a 5-gram LSTM ,…”