Modeling long distance dependence in language: topic mixtures vs. dynamic cache models

By Rukmini Iyer, Mari Ostendorf. 1996

We investigate a new statistical language model which captures topic-related dependencies of words within and across sentences. First, we develop a sentence-level mixture language model that takes advantage of the topic constraints in a sentence or article. Second, we introduce topic-dependent dynamic cache adaptation techniques in the framework of the mixture model. Experiments with the static (or unadapted) mixture model on the 1994 WSJ task indicated a 21% reduction in perplexity and a 3-4% improvement in recognition accuracy over a general n-gram model. The static mixture model also improved recognition performance over an adapted n-gram model. Mixture adaptation techniques contributed a further 14% reduction in perplexity and a small improvement in recognition accuracy.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A bit of progress in language modeling (2001, background): “Sentence Mixture models (Iyer and Ostendorf, 1999; Iyer et al., 1994) make use of the observation that there are many different sentence types, and that making models for each type of sentence may be better than using one global model.”
Recurrent neural network language model adaptation with curriculum learning (2015, methodology): “Second, as has been pointed out by Iyer et al. (1994), Iyer and Ostendorf (1999), and, more recently, by Mikolov and Zweig (2012), one danger of adaptation models that build individual component models on data sub-sets is fragmentation.”
Two decades of statistical language modeling: where do we go from here? (2000, background): “This is usually done via linear interpolation, at either the word probability level or the sentence probability level [65].”
Statistical language model adaptation: review and perspectives (2004, methodology): “For an insightful review of mixture and cache language models (see Iyer and Ostendorf, 1999 ).”
Learning author-topic models from text corpora (2010, background): “…approaches to modeling document content based on the idea that the probability distribution over words in a document can be expressed as a mixture of topics, where each topic is a probability distribution over words [Blei et al. 2003; Hofmann 1999; Ueda and Saito 2003; Iyer and Ostendorf 1999].”
Exploiting latent semantic information in statistical language modeling (2000, background): “Then each document can be characterized by drawing from a (possibly large) set of topics, usually predefined from a hand-labeled hierarchy , which covers the relevant semantic domain [39], [63], [64].”
Bayesian Language Model Interpolation for Mobile Speech Input (2011, background): “One scenario is topic modeling, where parts of a corpus are labeled by pre-defined topics and sub-language models are built on these topics and used in interpolation [8, 9].”
Segmenting DNA sequence into 'words' based on statistical language model (2012, background): “More research works about the adaptation of n-gram models could refer to (22,23).”
Query Expansion with Locally-Trained Word Embeddings (2016, methodology): “A topic-dependent model may be combined with the global model using linear interpolation (Iyer and Ostendorf, 1999)”