A Dynamic Language Model for Speech Recognition

By Frederick Jelinek, Bernard Mrialdo, Salim Roukos, Martin J. Strauss. 1991

In the case of a trigram language model, the probability of the next word conditioned on the previous two words is estimated from a large corpus of text. The resulting static trigram language model (STLM) has fixed probabilities that are independent of the document being dictated. To improve the language model (LM), one can adapt the probabilities of the trigram language model to match the current document more closely. The partially dictated document provides significant clues about what words are more likely to be used next. Of many methods that can be used to adapt the LM, we describe in this paper a simple model based on the trigram frequencies estimated from the partially dictated document. We call this model a cache trigram language model (CTLM) since we are caching the recent history of words. We have found that the CTLM reduces the perplexity of a dictated document by 23%. The error rate of a 20,000-word isolated word recognizer decreases by about 5% at the beginning of a document and by about 24% after a few hundred words.

Subfields: Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Speech & Audio

Citations in the corpus

Unbounded cache model for online language modeling with open vocabulary (2017, background): “Jelinek et al. further extended this strategy [29] into a smoothed trigram language model, reporting a reduction in both perplexity and word error rates.”
Improving Neural Language Models with a Continuous Cache (2016, methodology): “These models were further extended by Jelinek et al. (1991) into a smoothed trigram language model, reporting reduction in both perplexity and word error rates.”
Visualizing and Understanding Recurrent Networks (2015, background): “This limitation is related to the improvements seen in “dynamic evaluation” Mikolov (2012); Jelinek et al. (1991) of recurrent language models, where an RNN is allowed to train on the test set characters during evaluation as long as it sees them only once.”
A bit of progress in language modeling (2001, background): “This observation is the basis of caching (Kuhn, 1988; Kuhn and De Mori, 1990; Kuhn and De Mori, 1992; Kupiec, 1989; Jelinek et al., 1991).”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, methodology): “More speciﬁc to the LM literature, prior work proposed ways of designing LMs that efﬁciently adapt their knowledge to continuous streams of new information (Jelinek et al., 1991; Wang et al., 2008; Goyal et al., 2009; Osborne et al., 2014; Yogatama et al., 2014 et al., 2019; Sun et al., 2020).”
Adaptive Language Modeling Using Minimum Discriminant Estimation (1992, methodology): “In [4] a trigram language was estimated from the cache and interpolated with the static trlgram model to yield about 20% lower perplexity and from 5% to 25% lower recognition error rate on documents ranging in length from 100 to 800 words.”
Pitfalls of Static Language Modelling (2021, background): “…prior work has proposed ways of designing language models that can efﬁciently adapt their knowledge to continuous streams of new information (Jelinek et al., 1991; Wang et al., 2008; Goyal et al., 2009; Osborne et al., 2014; Yogatama et al., 2014, inter alia )—often known as streaming…”
Recurrent neural network language model adaptation with curriculum learning (2015, background): “Recently, RNNLMs have been demonstrated to outperform n-gram language models for speech recognition (Mikolov et al., 2010). Their superior capabilities rely on two mechanisms. First, RNNLMs map the discrete word-based vocabulary into a continuous space. As a result, the model can exploit word sequen”
Shrinking Exponential Language Models (2009, methodology): “Jelinek et al. (1991) describe its use for combining a cache language model and static language model.”
Statistical Language Models Based on Neural Networks (2012, methodology): “As the cache models provide truly significant improvements in perplexity (sometimes even more than 20%), there exists a large number of more refined techniques that can capture the same patterns as the basic cache models - for example, various topic models, latent semantic analysis based models [3],”
Two decades of statistical language modeling: where do we go from here? (2000, background): “[61, 62] report reduction in perplexity, and [63] also reports reduction in recognition error rate.”