Joint and Conditional Estimation of Tagging and Parsing Models

By Mark Johnson. 2001

66 citations recorded by the source corpus.

This paper compares two different ways of estimating statistical language models. Many statistical NLP tagging and parsing models are estimated by maximizing the (joint) likelihood of the fully-observed training data. However, since these applications only require the conditional probability distributions, these distributions can in principle be learnt by maximizing the conditional likelihood of the training data. Perhaps somewhat surprisingly, models estimated by maximizing the joint were superior to models estimated by maximizing the conditional, even though some of the latter models intuitively had access to "more information".

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Noah A. Smith
2. Yoav Goldberg
3. Luke S. Zettlemoyer
4. Percy Liang
5. James Henderson
6. Ivan Titov
7. Daniel Klein
8. Christopher D. Manning

In Natural Language Processing

1. Noah A. Smith
2. Daniel Klein
3. Michael Collins
4. Christopher D. Manning
5. Dan Roth
6. James Henderson
7. Yoav Goldberg
8. Percy Liang

Citations in the corpus

• Recurrent Neural Network Grammars (2016, neutral): “Surprisingly—although in line with previous parsing results showing the effectiveness of generative models (Henderson, 2004; Johnson, 2001)— parsing with the generative model obtains signifi-cantly better results than parsing with the discriminative model.”
• Discriminative Training of a Neural Network Statistical Parser (2004, methodology): “While both (Johnson, 2001) and (Klein and Manning, 2002) propose models which use the parameters of the generative model but train to optimize a discriminative criteria, neither proposes training algorithms which are computationally tractable enough to be used for broad coverage parsing.”
• Contrastive Estimation: Training Log-Linear Models on Unlabeled Data (2005, background): “Klein and Manning (2002) argue for CL on grounds of accuracy, but see also Johnson (2001).”
• Iterative parameter mixing for distributed large-margin training of structured predictors for natural language processing (2015, methodology): “This concern prompted a movement in NLP to training using conditional likelihood (Collins, 2000; Johnson, 2001; Charniak & Johnson, 2005; Clark & Curran, 2007; Finkel et al., 2008).”
• Structured Prediction Models via the Matrix-Tree Theorem (2007, methodology): “Global log-linear training has been used in the context of PCFG parsing (Johnson, 2001).”