Adaptive Statistical Language Modeling; A Maximum Entropy Approach

By R. Rosenfeld. 1994

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

A bit of progress in language modeling (2001, methodology): “Maximum entropy models (Darroch and Ratcliff, 1972) have received a fair amount of attention since their introduction for language modeling by Rosenfeld (1994).”
Classes for fast maximum entropy training (2001, methodology): “[1] gives the classic Generalized Iterative Scaling algorithm, although in a form suitable for joint probabilities, as opposed to the conditional probabilities given here, and is somewhat dense; [2] is a classic introduction to the use of maximum entropy models for language modeling, but despite the”
Statistical language modeling using the CMU-cambridge toolkit (1997, methodology): “Linear Interpolation of Models In order to facilitate the combination of language models, the toolkit also includes a program [15, Appendix B] for calculating maximum likelihood weights for a set of models by use of the expectation maximisation (EM) al-gorithm [4].”
Sparse Non-negative Matrix Language Modeling (2015, methodology): “The type of long-distance features that we incorporate into our SNMLMs are skip-grams (Huang et al., 1993; Ney et al., 1994; Rosenfeld, 1994), which can effectively capture dependencies across longer contexts.”
Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language modeling (2000, background): “As explained by Rosenfeld(1994), this sum may be simpli ed by dividing the vocabulary into words whichdo not have bigram, trigram or topic-dependent constraints, and words whichdo.”
Data Mining for Web Personalization (2007, methodology): “Jin et al. [54] proposed a more robust approach based on hidden variable models in which users’ navigational data and the content features associated with items are seamlessly integrated using a maximum entropy approach [111].”
Recurrent neural network language model adaptation with curriculum learning (2015, methodology): “In this paper, we investigate language model adaptation for RNNLMs, and specifically address two central challenges for language model adaptation, within-domain adaptation and limited-data within-domain adaptation, originally identified by Rosenfeld (1994) and explained later in depth.”
A maximum entropy approach to adaptive statistical language modelling (1996, methodology): “See [Rosenfeld 94b] for a more detailed exposition.”
Part-of-Speech Tagging Using Decision Trees (1998, methodology): “Other works that can be placed in this class are those of (Schmid 94a) which performs energy{function optimization using neural nets and (Rosenfeld 94) which has applied a Maximum Entropy Approach to POS tagging.”
Fast Evaluation of Connectionist Language Models (2009, background): “The measure of the quality of a estimated LM is usually the perplexity of a test set which can be intuitively interpreted as the geometric mean of the branch-out factor of the language [2, 11 ]: a language with perplexity x has roughly the same difficulty as another language in which every word can ”
Statistical Language Models Based on Neural Networks (2012, methodology): “Maximum entropy models [64] allow easy integration of any information source in the probabilistic model.”
Word Clustering Using PLSA Enhanced with Long Distance Bigrams (2010, methodology): “In addition, language models combining the baseline bigram with trigger pairs [9] at various histories (d = 2, 3, 4 and d = 6) were also incorporated in the PLSA-based clustering algorithm for comparison purposes.”
Developing a fuzzy TOPSIS approach based on subjective weights and objective weights (2009, methodology): “Later research has applied his measure to a wide range of applications including spectral analysis (Burg, 1967), language modeling (Rosenfeld, 1994) and economics (Golan, Judge, & Miller, 1996).”
Two decades of statistical language modeling: where do we go from here? (2000, methodology): “For example, in [8], [83], conventional-grams, distance-2 -grams, and long-distance word pairs (“triggers”) were encoded as features and resulted in up to 39% perplexity”
Structured language modeling (2000, background): “The two interesting enhancements, facilitated by maximum entropy estimation methodology, have been the use of triggers (Rosenfeld, 1994) or of singular value decomposition (Bellegarda, 1997) (either of which dynamically identify the topic of discourse) in combination with n-gram models.”
A Closer Look at Skip-gram Modelling (2006, methodology): “Skip-grams have been used many different ways in language modelling but often in conjunction with other modelling techniques or for the goal of decreasing perplexity (Goodman, 2001; Rosenfeld, 1994; Ney et al., 1994; Siu and Ostendorf, 2000).”
Word Clustering Using PLSA Enhanced with Long Distance Bigrams (2010, methodology): “In addition, language models combining the baseline bigram with trigger pairs [9] at various histories (d = 2, 3, 4 and d = 6) were also incorporated in the PLSA-based clustering algorithm for comparison purposes.”