Modeling long distance dependence in language: topic mixtures versus dynamic cache models

By Rukmini Iyer, Mari Ostendorf. 1999

Standard statistical language models use n-grams to capture local dependencies, or use dynamic modeling techniques to track dependencies within an article. In this paper, we investigate a new statistical language model that captures topic-related dependencies of words within and across sentences. First, we develop a topic-dependent, sentence-level mixture language model which takes advantage of the topic constraints in a sentence or article. Second, we introduce topic-dependent dynamic adaptation techniques in the framework of the mixture model, using n-gram caches and content word unigram caches. Experiments with the static (or unadapted) mixture model on the North American Business (NAB) task show a 21% reduction in perplexity and a 3-4% improvement in recognition accuracy over a general n-gram model, giving a larger gain than that obtained with supervised dynamic cache modeling. Further experiments on the Switchboard corpus also showed a small improvement in performance with the sentence-level mixture model. Cache modeling techniques introduced in the mixture framework contributed a further 14% reduction in perplexity and a small improvement in recognition accuracy on the NAB task for both supervised and unsupervised adaptation.

Subfields: Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “Other adaptive language models have been proposed in the past: Kneser and Steinbiss [35] and, Iyer and Ostendorf [26] dynamically adapt the parameters of their model to recent history using different weight interpolation schemes.”
Two decades of statistical language modeling: where do we go from here? (2000, background): “1. Clustering the training corpus along the dimension of variability, say, topic (e.g. [ 65 ]).”
Recurrent neural network based language model personalization by social network crowdsourcing (2013, methodology): “In traditional N-gram models, the two corpora are interpolated; various methods are used to estimate their interpolation weights [35, 36, 37, 38].”