A maximum entropy approach to adaptive statistical language modelling

By R. Rosenfeld. 1996

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

Pointer Sentinel Mixture Models (2016, methodology): “Rosenfeld (1996) uses a maximum entropy model to combine a variety of information sources to improve language modeling on news text and speech.”
Unbounded cache model for online language modeling with open vocabulary (2017, background): “r, smoothed N-gram models [32, 34] have been the dominant type of models historically, achieving good performance in practice [44]. While the use of parametric models for language modeling is not new [48], their superiority has only been established with the recent emergence of neural networks [7, 4”
Improving Neural Language Models with a Continuous Cache (2016, background): “Parametrized alternatives are either maximum entropy language models (Rosenfeld, 1996), feedforward networks (Bengio et al., 2003) or recurrent networks (Mikolov et al., 2010).”
Fast and Robust Neural Network Joint Models for Statistical Machine Translation (2014, methodology): “far too sparse for standard probability models such as Kneser-Ney back-off (Kneser and Ney, 1995) or Maximum Entropy (Rosenfeld, 1996).”
Domain Adaptation: Learning Bounds and Algorithms (2009, background): “…Blitzer et al., 2007; Jiang & Zhai, 2007; Chelba & Acero, 2006; Daumé III & Marcu, 2006), speech processing (Legetter & Woodland, 1995; Gauvain & Chin-Hui, 1994; Pietra et al., 1992; Rosenfeld, 1996; Jelinek, 1998; Roark & Bacchiani, 2003), computer vision (Martı́nez, 2002), and many other areas.”
Multiple Source Adaptation and the Rényi Divergence (2009, background): “ther access a large set of unlabeled points from a target domain. This problem arises in a variety of natural language processing tasks such parsing, statistical language modeling, text classiﬁcation [15, 7, 16, 9, 10, 4, 6], or speech processing [11, 8, 14] and computer vision [13] tasks, as well a”
Offline recognition of unconstrained handwritten texts using HMMs and statistical language models (2004, background): “proposed in [ 26 ]) and is based on the inverse proportionality between r and nr. This property of many”
Two decades of statistical language modeling: where do we go from here? (2000, methodology): “For example, in [8], conventional ngrams, distance-2 ngrams, and long distance word pairs (“triggers”) were encoded as features, and resulted in up to 39% perplexity reduction and up to 14% speech recognition word error rate reduction over the trigram baseline.”
Towards better integration of semantic predictors in statistical language modeling (1998, unknown): “[ 8 ], [6]). In previous work, we [1] and others [2], [5] have suggested the use of Latent Semantic Analysis (LSA) [3] as a model of semantic knowledge to be applied to ASR.”
Generalizing and Hybridizing Count-based and Neural Language Models (2016, background): “Also conceptually similar is work on adaptation of n-gram LMs, which start with n-gram probabilities (Della Pietra et al., 1992; Kneser and Steinbiss, 1993; Rosenfeld, 1996; Iyer and Ostendorf, 1999) and adapt them based on the distribution of the current document, albeit in a linear model.”
Recurrent neural network language model adaptation with curriculum learning (2015, background): “Rosenfeld (1996) showed that maximum entropy language models, using trigger and n-gram features, achieve a significant improvement over n-gram language models in terms of perplexity and word error rate.”
A Bayesian Mixed Effects Model of Literary Character (2014, methodology): “Maximum entropy approaches to language modeling have been used since Rosenfeld (1996) to incorporate long-distance information, such as previously-mentioned trigger words, into n -gram language models.”
On the computational basis of learning and cognition: Arguments from LSA (2002, methodology): “…on words-to-following-words association matrices, as is done in the HAL model of Lund and Burgess (1996), the SP model of Simon Dennis (2001), and the language models used in automatic speech recognition (e.g. Rabiner, 1969; Rosenfeld,1996), in addition to the word-to-passage equations of LSA.”
Domain Adaptation with Multiple Sources (2008, background): “This set-up and the type of solutions just described have been in fact explored extensively in applications [8, 9, 11, 13–15].”
A Sparse Plus Low Rank Maximum Entropy Language Model (2012, background): “The standard maximum entropy language model [2] has the form”
Probabilistic Top-Down Parsing and Language Modeling (2001, background): “work There have been attempts to jump over adjacent words to words farther back in the leftcontext, without the use of dependency links or syntactic structure, for example Saul and Pereira (1997) and Rosenfeld (1996; 1997). We will focus our very brief review, however, on those which use grammars or”
Adaptive weighing of context models for lossless data compression (2005, background): “Model mixing and sparse word contexts are based on offline language models for speech recognition (Rosenfeld 1996).”