Contrastive Estimation: Training Log-Linear Models on Unlabeled Data

By Noah A. Smith, Jason Eisner. 2005

381 citations recorded by the source corpus.

Conditional random fields (Lafferty et al., 2001) are quite effective at sequence labeling tasks like shallow parsing (Sha and Pereira, 2003) and named-entity extraction (McCallum and Li, 2003). CRFs are log-linear, allowing the incorporation of arbitrary features into the model. To train on unlabeled data, we require unsupervised estimation methods for log-linear models; few exist. We describe a novel approach, contrastive estimation. We show that the new technique can be intuitively understood as exploiting implicit negative evidence and is computationally efficient. Applied to a sequence labeling problem---POS tagging given a tagging dictionary and unlabeled text---contrastive estimation outperforms EM (with the same feature set), is more robust to degradations of the dictionary, and can largely recover by modeling additional features.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Optimization & Training

Citations in the corpus

Natural Language Processing (Almost) from Scratch (2011, background): “Smith and Eisner (2005) also propose a contrastive criterion which estimates the likelihood of the data conditioned to a “negative” neighborhood.”
Reasoning about Pragmatics with Neural Listeners and Speakers (2016, background): “Note that this objective can also be viewed as the contrastive loss described by Smith and Eisner (2005), where it serves as an approximation to the likelihood objective that encourages L 0 to prefer r i to every other possible referent simultaneously.”
Learning to Make Analogies by Contrasting Abstract Relational Structure (2019, background): “This underlines the fact that, for established learning algorithms involving negative examples such as (noise) contrastive estimation (Smith & Eisner, 2005; Gutmann & Hyv¨arinen, 2010) or negative sampling (Mikolov et al., 2013), the way in which negative examples are selected can be critical 3 .”
Structured Prediction Models via the Matrix-Tree Theorem (2007, background): “, 1970), contrastive estimation (Smith and Eisner, 2005), training algorithms for CRFs (Lafferty et al.”
Successive Prompting for Decomposing Complex Questions (2022, methodology): “To deter the model from learning incorrect steps, we use contrastive estimation (Smith and Eisner, 2005).”
A fully Bayesian approach to unsupervised part-of-speech tagging (2007, methodology): “Recently, several new model-based approaches have improved performance on a variety of tasks (Klein and Manning, 2002; Smith and Eisner, 2005).”
The effects of part–of–speech tagging on text–to–speech synthesis for resource–scarce languages (2010, methodology): “Contrastive Estimation (CE) [78] is an alternative training method to EM for HMMs.”
Guiding Semi-Supervision with Constraint-Driven Learning (2007, methodology): “This was used, for example, by (Thelen and Riloff, 2002; Collins and Singer, 1999) in information extraction, and by (Smith and Eisner, 2005) in POS tagging.”
Modeling Relations and Their Mentions without Labeled Text (2010, neutral): “Contrastive estimation [25] is another related approach.”
Learning Entity Representation for Entity Disambiguation (2013, background): “…is deﬁned as negative log of softmax function: P Finally, we seek to minimize the following training objective across all training instances: The loss function is closely related to contrastive estimation (Smith and Eisner, 2005), which deﬁnes where the positive example takes probability mass from.”
Collapsed Variational Bayesian Inference for Hidden Markov Models (2013, methodology): “Following Smith and Eisner (2005), we randomly select 1K unlabelled sentences from the treebank for the training data.”
Learning Entailment Relations by Global Graph Structure Optimization (2012, background): “In the spirit of “contrastive estimation” (Smith and Eisner 2005), we prefer to generate negative examples that are “hard,” that is, negative examples that, although not entailing, are still semantically similar to positive examples and thus focus the classiﬁer’s attention on determining the…”
Distributional Representations for Handling Sparsity in Supervised Sequence-Labeling (2009, methodology): “…use Hidden Markov Models (HMMs) as the main example in the discussion and as the LVLMs in our experiments, but the smoothing technique can be generalized to other forms of LVLMs, such as factorial HMMs and latent variable maximum entropy models (Ghahramani and Jordan, 1997; Smith and Eisner, 2005).”
Generalized Expectation Criteria for Semi-Supervised Learning with Weakly Labeled Data (2010, background): “In contrastive estimation (Smith and Eisner, 2005), EM is performed over a restricted loglikelihood function, where instead of L(θ) = ∑i log p(xi;θ), the contrastive estimation log-likelihood function is LCE(θ) = ∑i log p(xi|N (xi);θ).”
Recursive deep learning for natural language processing and computer vision (2014, background): “This is similar to implicit negative evidence in contrastive estimation (Smith and Eisner, 2005).”