A bit of progress in language modeling

By Joshua Goodman. 2001

740 citations recorded by the source corpus.

In the past several years, a number of different language modeling improvements over simple trigram models have been found, including caching, higher-order n -grams, skipping, interpolated Kneser?Ney smoothing, and clustering. We present explorations of variations on, or of the limits of, each of these techniques, including showing that sentence mixture models may have more potential. While all of these techniques have been studied separately, they have rarely been studied in combination. We compare a combination of all techniques together to a Katz smoothed trigram model with no count cutoffs. We achieve perplexity reductions between 38 and 50% (1 bit of entropy), depending on training data size, as well as a word error rate reduction of 8.9%. Our perplexity reductions are perhaps the highest reported compared to a fair baseline.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

One billion word benchmark for measuring progress in statistical language modeling (2013, methodology): “This follows the work of Goodman (2001a), who explored performance of various language modeling techniques when applied to large data sets.”
Scaling Laws for Neural Language Models (2020, background): “Some early [BB01, Goo01] work found power-law scalings between performance and dataset size.”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “All tools developed in the process are available on GitHub.11 Obtaining the Source Data The first step involved obtaining the data for all of the text data sources identified in Section 3.1.2, which consisted of a combination of downloading and extracting the text field from a variety of NLP dataset”
Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “Using the chain rule, the probability assigned to a sequence of words x 1 , . . . , x T can be factorized as This conditional probability is traditionally approximated with non-parametric models based on counting statistics [20].”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, background): “…of the observations by setting θuusert ,u system t−1 ,u user t−2,...,u system t−n−1 equal to (a constant times) the number of occurrences of each corresponding n-gram , or on a related objective function which encourages smoothness and therefore reduces data sparsity for larger n’s (Goodman, 2001).”
Learning Longer Memory in Recurrent Neural Networks (2014, neutral): “This has been widely studied in the past with approaches taking their roots in a variety of ﬁelds (Goodman, 2001b; Young et al., 1997; Koehn et al., 2007).”
Augmenting Self-attention with Persistent Memory (2019, methodology): “Solutions are to either replace the softmax loss with an approximation [12, 29], to sample from the vocabulary during training [5, 21] or to include subword units [34].”
What Context Features Can Transformer Language Models Use? (2021, methodology): “Subsequent models improved on these by allowing the use of skip-grams, caches, and feature-based models (Goodman, 2001; Bengio et al., 2003).”
Composition-based on-the-fly rescoring for salient n-gram biasing (2015, background): “Several improvements to the original formulation, including a decay function [9] and generalization to higher order n-grams [10] have made this a potentially effective mode of personalization.”
Statistical Language Models Based on Neural Networks (2012, methodology): “The simple reason is that in the real-world applications (such as speech recognizers), there is a strong positive correlation between perplexity of involved language model and the system’s performance [24].”
A Pruned Rnnlm Lattice-Rescoring Algorithm for Automatic Speech Recognition (2018, background): “In recent years, with the accessibility of greater computing power, recurrent neural network language models (RNNLM) [1] have become possible and have quickly surpassed back-off n-gram models [2] in various language-related tasks.”
Neural network based language models for highly inflective languages (2009, background): “While larger networks perform better, it takes much more time to train them - training 40 60 network is two times more time consuming than training 20 30 network.”