An Empirical Study of Smoothing Techniques for Language Modeling

By Stanley F. Chen, Joshua Goodman. 1996

We present an extensive empirical comparison of several smoothing techniques in the domain of language modeling, including those described by Jelinek and Mercer (1980), Katz (1987), and Church and Gale (1991). We investigate for the first time how factors such as training data size, corpus (e.g., Brown versus Wall Street Journal), and n-gram order (bigram versus trigram) affect the relative performance of these methods, which we measure through the cross-entropy of test data. In addition, we introduce two novel smoothing techniques, one a variation of Jelinek-Mercer smoothing and one a very simple linear interpolation technique, both of which outperform existing methods.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, background): “…et al., 2012) made much of WS2 advances possible, but perception, interaction, and social robotics bring new challenges which our current tools are not equipped to handle. with Good (1953), the weights of Kneser and Ney (1995); Chen and Goodman (1996), and the power-law distributions of Teh (2006).”
A Survey of Machine Learning for Big Code and Naturalness (2017, methodology): “Instead, n-gram models use smoothing methods [40] as a principled way for assigning probability to unseen n-grams by extrapolating information fromm-grams (m < n).”
Learning Visual N-Grams from Web Data (2016, methodology): “2A good overview of these techniques is given in [8, 22].”
Librispeech: An ASR corpus based on public domain audio books (2015, methodology): “Modified Kneser-Ney smoothed 3- and 4-grams [21, 22] are trained.”
Language Modeling with Gated Convolutional Networks (2016, background): “Recently, neural networks (Bengio et al., 2003; Mikolov et al., 2010; Jozefowicz et al., 2016) outperform classical n-gram language models (Kneser & Ney, 1995; Chen & Goodman, 1996).”
Character-Aware Neural Language Models (2015, methodology): “A language model is formalized as a probability distribution over a sequence of strings (words), and traditional methods usually involve making an n-th order Markov assumption and estimating n-gram probabilities via counting and subsequent smoothing (Chen and Goodman 1998).”
Effective Self-Training for Parsing (2006, methodology): “As all distributions are conditioned with ﬁve or more features, they are all heavily backed off using Chen back-off (the average-count method from Chen and Goodman (1996)).”
Visualizing and Understanding Recurrent Networks (2015, methodology): “n-gram: An unpruned (n + 1)-gram character-level language model using modified KneserNey smoothing (3).”
A bit of progress in language modeling (2001, background): “This is useful in a large variety of areas including speech recognition, optical character recognition, handwriting recognition, machine translation, and spelling correction (Church, 1988; Brown et al., 1990; Hull, 1992; Kernighan et al., 1990; Srihari and Baltus, 1992).”
Montreal Neural Machine Translation Systems for WMT’15 (2015, positive): “For the other language pairs, we tried reranking n-best lists with 5-gram language models (Chen and Goodman., 1998).”
Direct Output Connection for a High-Rank Language Model (2018, methodology): “Their method outperformed the Kneser-Ney smoothed 5-gram language model (Kneser and Ney, 1995; Chen and Goodman, 1996).”
Data Noising as Smoothing in Neural Network Language Models (2017, methodology): “However, since “Francisco” and “York” typically follow “San” and “New”, respectively, they should not have high probability in the proposal distribution as they might if we use unigram frequencies (Chen & Goodman, 1996).”
BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2019, methodology): “decade, we have seen two major paradigm shifts in the state of the art for language modeling. The ﬁrst major shift for language modeling was the movement from statistical methods based on n-grams (seeChen and Goodman, 1999) to neural methods such as LSTMs (Sundermeyer et al.,2012), which directly op”
An MDP-Based Recommender System (2002, neutral): “Details of these and other methods are given in (e.g., Chen & Goodman, 1998).”
Compositional Morphology for Word Representations and Language Modelling (2014, methodology): “· In contrast, discrete n -gram models are estimated by smoothing and backing off over empirical distributions (Kneser & Ney, 1995; Chen & Goodman, 1998).”