Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art

By Veselin Stoyanov, Nathan Gilbert, Claire Cardie, Ellen Riloff. 2009

167 citations recorded by the source corpus.

We aim to shed light on the state-of-the-art in NP coreference resolution by teasing apart the differences in the MUC and ACE task definitions, the assumptions made in evaluation methodologies, and inherent differences in text corpora. First, we examine three subproblems that play a role in coreference resolution: named entity recognition, anaphoricity determination, and coreference element detection. We measure the impact of each subproblem on coreference resolution and confirm that certain assumptions regarding these subproblems in the evaluation methodology can dramatically simplify the overall task. Second, we measure the performance of a state-of-the-art coreference resolver on several classes of anaphora and use these results to develop a quantitative measure for estimating coreference resolution performance on new data sets.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Easy Victories and Uphill Battles in Coreference Resolution (2013, background): “However, there remain a few natural features that we omitted in order to keep the system as simple as possible, since they were orthogonal to the discussion of data-driven versus heuristic-driven features and do not target world knowledge.”
What Are You Talking About? Text-to-Image Coreference (2014, methodology): “3We use a version of the B3 metric called B3all, defined by [30] that can handle (and suitably penalize) unmatched GT and predicted mentions.”
Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, methodology): “Interms of metrics, most intrinsic evaluation is carried out using metrics like MUC (Vilain et al. 1995), ACE (Mitchell et al. 2005), B 3 (Bagga and Baldwin 1998; Stoyanov et al. 2009), or CEAF (Luo 2005) (see also (Cai and Strube 2010) for additional discussion and variants).”
Supervised Noun Phrase Coreference Research: The First Fifteen Years (2010, methodology): “To apply these scorers to automatically extracted NPs, different methods have been proposed (see Rahman and Ng (2009) and Stoyanov et al. (2009)).”
An Active Learning Approach to Coreference Resolution (2015, methodology): “After building a pairwise model to judge coreferentiality or not, these judgments are aggregated to obtain final coreference chains via some heuristics [Stoyanov et al., 2009] or yet another model [Finkel and Manning, 2008].”
Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric (2016, background): “Another problem with CEAFe, mentioned by Stoyanov et al. (2009), is that it weights entities equally regardless of their sizes.”
A Joint Model for Entity Analysis: Coreference, Typing, and Linking (2014, methodology): “We use the train/test split from Stoyanov et al. (2009), Haghighi and Klein (2010), and Bansal and Klein (2012).”
Coreference Resolution with World Knowledge (2011, methodology): “To address this problem, we set the recall and precision of a twinless NP to zero, regardless of whether the NP appears in the key or the response.”
CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes (2011, background): “Given the space constraints, we refer the reader to Stoyanov et al. (2009) for a detailed treatment of the issue.”
Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation (2014, methodology): “Coining the term twinless mentions for those mentions that are either spurious or missing from the predicted mention set, Stoyanov et al. (2009) proposed two variations to B 3 —”
Coreference Semantics from Web Features (2012, methodology): “We ﬁrst describe a strong baseline consisting of the mention-pair model of the Reconcile system (Stoyanov et al., 2009; Stoyanov et al., 2010) us-ing a decision tree (DT) as its pairwise classiﬁer.”
A Constrained Latent Variable Model for Coreference Resolution (2013, methodology): “This strategy has been successful and commonly used in coreference resolution (Ng and Cardie, 2002; Bengtson and Roth, 2008; Stoyanov et al., 2009).”
Evaluation Metrics For End-to-End Coreference Resolution Systems (2010, methodology): “CEAF orig reports very low precision for system mentions (see also Stoyanov et al. (2009)).”
Evaluation of named entity coreference (2019, background): “Detection of named mentions can be done with high accuracy by named entity recognition systems (Stoyanov et al., 2009) and the matching of names can also be done accurately via string matching (Wacholder et al., 1997; Wick et al., 2009).”
CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes (2012, background): “This has made it hard to gauge the improvement in algorithms over the years (Stoyanov et al., 2009), or to determine which particular areas require further attention.”
Robustness in Coreference Resolution (2020, result): “As shown in the literature (Stoyanov et al., 2009; Recasens & Vila, 2010), corpus parameters should be in agreement for the fair comparison of coreference approaches.”
A Multi-Pass Sieve for Coreference Resolution (2010, methodology): “To address this issue, this pass implements several features that must all be matched in order to yield a link: Cluster head match – the mention head word matches any head word in the antecedent cluster.”