A Model-Theoretic Coreference Scoring Scheme

By M. Vilain, John D. Burger, J. Aberdeen, D. Connolly, Lynette Hirschman. 1995

This note describes a scoring scheme for the coreference task in MUC6. It improves on the original approach by: (1) grounding the scoring scheme in terms of a model; (2) producing more intuitive recall and precision scores; and (3) not requiring explicit computation of the transitive closure of coreference. The principal conceptual difference is that we have moved from a syntactic scoring model based on following coreference links to an approach defined by the model theory of those links.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Toward Gender-Inclusive Coreference Resolution (2019, background): “.2013], six times as many in News Commentaries [Tiedemann2012], and fifty times as many in Hong Kong Laws corpus. In terms of metrics, most intrinsic evaluation is carried out using metrics like MUC [Vilain et al. 1995], ACE [Mitchell et al.2005], B3 [Bagga and Baldwin1998;Stoyanov et al.2009], or C”
Deep Reinforcement Learning for Mention-Ranking Coreference Models (2016, methodology): “Although our system evaluation also includes the MUC (Vilain et al., 1995) and CEAFφ4 (Luo, 2005) metrics, we do not incorporate them into the loss because MUC has the flaw of treating all errors equally and CEAFφ4 is slow to compute.”
Deep Read: A Reading Comprehension System (1999, methodology): “Using the MUC coreference scoring algorithm (see Vilain et al. 1995), this had a precision of 77% and a recall of 18%.”
Modeling Local Coherence: An Entity-Based Approach (2005, methodology): “2 When evaluating the output of coreference algorithms, performance is typically measured using a model-theoretic scoring scheme proposed in Vilain et al. (1995). The scoring algorithm computes the recall error by taking each equivalence class S in the gold standard and determining the number of cor”
Solving Hard Coreference Problems (2019, methodology): “Evaluation Metrics: When evaluating on the full datasets of ACE and OntoNotes , we use the widely recoginzed metrics MUC (Vilain et al., 1995), BCUB (Bagga and Baldwin, 1998), Entity-based CEAF (CEAF e ) (Luo, 2005) and their average.”
Dynamic Entity Representations in Neural Language Models (2017, methodology): “It computes three commonly used evaluation measures MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF e (Luo, 2005).”
What Are You Talking About? Text-to-Image Coreference (2014, methodology): “As shown by previ- ous work [20], the MUC metric is somewhat biased towards large clusters, whereas the B3 metric is biased towards singleton clusters.”
Entity-Centric Coreference Resolution with Model Stacking (2015, methodology): “For m , the performance metric determining the action costs, we use a linear combination of the B 3 (Bagga and Baldwin, 1998) and MUC (Vilain et al., 1995) metrics, which are both commonly used for evaluating coreference systems.”
Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, methodology): “In terms of metrics, most intrinsic evaluation is carried out using metrics like MUC (Vilain et al. 1995), ACE (Mitchell et al.”
Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric (2016, methodology): “Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 632–642, Berlin, Germany, August 7-12, 2016. c©2016 Association for Computational Linguistics”
Evaluation of named entity coreference (2019, methodology): “Shared tasks on coreference (at CoNLL-2011 and 2012 (Pradhan et al., 2014) ) use the average of three F1 scores as their ofﬁcial evaluation: MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998) and CEAFE (Luo, 2005).”
RelaxCor Participation in CoNLL Shared Task on Coreference Resolution (2011, methodology): “The results obtained with MUC scorer show an improvement of R ELAX C OR ’s recall, a feature that needed improvement given the previous published results with a MUCs recall remarkably low (Sapena et al., 2010b).”
Improving Coreference Resolution by Learning Entity-Level Distributed Representations (2016, methodology): “Although our system evaluation also includes the MUC (Vilain et al., 1995) and CEAFφ4 (Luo, 2005) metrics, we do not incorporate them into the loss because MUC has the flaw of treating all errors equally and CEAFφ4 is slow to compute.”
Survey Article: Inter-Coder Agreement for Computational Linguistics (2008, unknown): “This proposal is in line with the methods developed to evaluate anaphora resolution systems (Vilain et al. 1995).”
Global joint models for coreference resolution and named entity classification (2009, background): “Three primary metrics have been proposed for evaluating coreference performance: (i) the link based muc metric (Vilain et al., 1995), (ii) the mention based b3 metric (Bagga and Baldwin, 1998), and (iii) the entity based ceaf metric (Luo, 2005).”
A Multi-Pass Sieve for Coreference Resolution (2010, methodology): “Table 4 shows the number of incorrect pair-wise links generated by our system on the MUC6-TEST corpus.”
Understanding the Value of Features for Coreference Resolution (2008, methodology): “MUC F-Score We also provide results using the official MUC scoring algorithm (Vilain et al., 1995).”