A Maximum Entropy Approach to Natural Language Processing

By Adam L. Berger, Stephen Della Pietra, V. D. Pietra. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Normalized Entropy (Berger et al., 1996), a measure of diversity in probability distribution normalized by number of tokens.”
From captions to visual concepts and back (2014, methodology): “Following the definition of an ME LM [2], the word probability conditioned on preceding words and remaining objects can be written as:”
A fast and simple algorithm for training neural probabilistic language models (2012, background): “1 can describe both neural and maximum entropy language models (Berger et al., 1996).”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, positive): “In supervised learning, we can search for the model with maximum entropy subject to constraints on empirical statistics, which naturally gives rise to maximum likelihood in log-linear models (see (Berger et al., 1996) for a review).”
Machine Learning Models that Remember Too Much (2017, background): “Support Vector Machines (SVM) [17] and logistic regression (LR) are popular for classification tasks such as text categorization [35] and other natural language processing problems [8].”
Sequence-to-sequence neural net models for grapheme-to-phoneme conversion (2015, neutral): “Following [19,20], Eq.”
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation (2018, background): “The objective J ( q ; x, y ) in (1) is the canonical maximum entropy problem that one often encounters in deriving a max-ent model (Berger et al., 1996), which has the analytic solution: b b 2) is a fairly generic solution which is agnostic to the choice of the similarity measure s .”
Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency (2018, methodology): “In log-linear models, including both the original work on maximum-entropy models (Berger et al., 1996), and later work on conditional random fields (Lafferty et al.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “ that likely to be contained in the image captions. Multiple instance learning (MIL) [96] is used to train the model for learning discriminative visual signatures of each word. A maximum entropy (ME) [12] language model is used for generating image captions from these words. Generated captions are r”
Shallow Parsing with Conditional Random Fields (2003, background): “…the likelihood with a spherical Gaussian weight prior (Chen and Rosenfeld, 1999): 3 Training Methods Lafferty et al. (2001) used iterative scaling algorithms for CRF training, following earlier work on maximum-entropy models for natural language (Berger et al., 1996; Della Pietra et al., 1997).”
Tuning support vector machines for biomedical named entity recognition (2002, methodology): “Support Vector Machines (SVMs) (Vapnik, 1995) and Maximum Entropy (ME) method (Berger et al., 1996) are powerful learning methods that satisfy such requirements, and are applied successfully to other NLP tasks (Kudo and Matsumoto, 2000; Nak-agawa et al., 2001; Ratnaparkhi, 1996).”
On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems (2016, background): “The second one is a maximum-entropy language model (Berger, Pietra, and Pietra 1996).”
Argumentation mining: the detection, classification and structure of arguments in text (2009, background): “First, the maximum entropy model, which adheres to the maximum entropy principle [2].”
Recent Advances of Large-Scale Linear Classification (2012, background): “Maximum Entropy (ME) [59] is a generalization of logistic regression for multi-class problems5 and a special case of conditional random fields [60] (see Section VIII-A).”
Discriminative Reranking for Natural Language Parsing (2000, methodology): “Equivalently, we again talk about minimizing the negative Note the similarity of equation (9) to the LogLoss function for classification in equation (4).”
Exploring Correlation of Dependency Relation Paths for Answer Extraction (2006, methodology): “(Berger et al., 1996) gave a good description of ME model.”