Softmax-Margin CRFs: Training Log-Linear Models with Cost Functions

By Kevin Gimpel, Noah A. Smith. 2010

161 citations recorded by the source corpus.

Subfields: Optimization & Training, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Information Extraction

Citations in the corpus

On Structured Prediction Theory with Calibrated Convex Surrogate Losses (2017, neutral): “, in conditional random fields [25], Φlog(f ,y) := log( ∑ ŷ∈Y exp fŷ)− fy, and their hybrids [38, 21, 22, 41].”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, background): “A canonical example is large margin structured prediction [32, 11], which explicitly uses supervision and considers an upper bound surrogate for task loss.”
Posterior calibration and exploratory analysis for natural language processing models (2015, background): “Another approach might be to directly constrain CalibErr = 0 during training, or try to reduce it as a training-time risk minimization or cost objective (Smith and Eisner, 2006; Gimpel and Smith, 2010; Stoyanov et al., 2011; Brümmer and Doddington, 2013).”
Easy Victories and Uphill Battles in Coreference Resolution (2013, methodology): “here plays an analogous role to the loss in structured max-margin objectives; incorporating it into a conditional likelihood objective is a technique called softmax-margin (Gimpel and Smith, 2010).”
Classical Structured Prediction Losses for Sequence to Sequence Learning (2017, methodology): “This paper revisits several objective functions that have been commonly used for structured prediction tasks in NLP (Gimpel and Smith, 2010) and apply them to a neural sequence to sequence model (Gehring et al., 2017b) ( § 2).”
Stability and Generalization in Structured Prediction (2016, background): “Thus, researchers (Gimpel and Smith, 2010; Hazan and Urtasun, 2010) have proposed a smooth alternative, based on the soft-max function.”
Quantifying Similarity between Relations with Fact Distribution (2019, background): “As shown in (Gimpel and Smith, 2010), Softmax-Margin Loss can be expressed as where R ( x ) denotes a structured output space for x , and (cid:104) x ( i ) , r ( i ) is i th example in training data.”
Human-Guided Learning for Probabilistic Logic Models (2018, methodology): “To achieve this, inspired by prior work of Gimpel and Smith (2010), we introduce a cost function c in the denominator of the log-likelihood function.”
A Constrained Latent Variable Model for Coreference Resolution (2013, methodology): “We then learn w by minimizing the regularized negative log-likelihood of the data, augmenting the partition function with a loss-based margin (Gimpel and Smith, 2010).”
A Joint Model for Entity Analysis: Coreference, Typing, and Linking (2014, methodology): “This technique, softmaxmargin, allows us to shape the distribution learned by the model and encourage the model to move probability mass away from outputs that are bad according to our loss functions (Gimpel and Smith, 2010).”
Multi-Task Cross-Lingual Sequence Tagging from Scratch (2016, methodology): “Therefore, we add a cost function between y and y (cid:48) based on the max-margin principle that high-cost tags y (cid:48) should be penalized more heavily (Gimpel and Smith, 2010).”
Minimum-Risk Training of Approximate CRF-Based NLP Systems (2012, methodology): “…resembles MLE but modiﬁes the denominator of (1) to In our experiments we compare against MLE training (which is common) and softmax-margin, which incorporates loss and which Gimpel and Smith (2010) show is either better or competitive when compared to other margin methods on an NLP task.”
SparseMAP: Differentiable Sparse Structured Inference (2018, unknown): “We evaluate the SparseMAP losses against the commonly used CRF and structured SVM losses.”
Learning Beam Search Policies via Imitation Learning (2018, unknown): “With unbounded beam capacity, we recover the softmax-margin loss of [25] for CRFs. weighted pairs (all) Reduces the problem of producing the correct ranking over the neighbors to n ( n − 1) / 2 weighted binary classiﬁcation problems.”
Syntactic Scaffolds for Semantic Structures (2018, methodology): “In early experiments, we found that incorporating a structured cost was beneficial; we do so by using a softmax-margin training objective (Gimpel and Smith, 2010), a “cost-aware” variant of log-likelihood:”
BRIO: Bringing Order to Abstractive Summarization (2022, background): “Among them, marginbased losses (Herbrich et al., 1999; Taskar et al., 2004; Gimpel and Smith, 2010), which require the model to assign higher probability to the better output, are a major category.”