On Coreference Resolution Performance Metrics

By Xiaoqiang Luo. 2005

676 citations recorded by the source corpus.

The paper proposes a Constrained Entity-Alignment F-Measure (CEAF) for evaluating coreference resolution. The metric is computed by aligning reference and system entities (or coreference chains) with the constraint that a system (reference) entity is aligned with at most one reference (system) entity. We show that the best alignment is a maximum bipartite matching problem which can be solved by the Kuhn-Munkres algorithm. Comparative experiments are conducted to show that the widely-known MUC F-measure has serious flaws in evaluating a coreference system. The proposed metric is also compared with the ACE-Value, the official evaluation metric in the Automatic Content Extraction (ACE) task, and we conclude that the proposed metric possesses some properties such as symmetry and better interpretability missing in the ACE-Value.

Subfields: Artificial Intelligence, Control and Systems Engineering, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Toward Gender-Inclusive Coreference Resolution (2019, methodology): “In terms of metrics, most intrinsic evaluation is carried out using metrics like MUC [Vilain et al. 1995], ACE [Mitchell et al. 2005], B 3 [Bagga and Baldwin 1998; Stoyanov et al. 2009], or CEAF [Luo 2005] (see also [Cai and Strube 2010] for additional discussion and variants).”
Easy Victories and Uphill Battles in Coreference Resolution (2013, methodology): “For each experiment, we report MUC (Vilain et al., 1995), B3 (Bagga and Baldwin, 1998), and CEAFe (Luo, 2005), as well as their average, the CoNLL metric.”
Solving Hard Coreference Problems (2019, methodology): “Evaluation Metrics: When evaluating on the full datasets of ACE and OntoNotes , we use the widely recoginzed metrics MUC (Vilain et al., 1995), BCUB (Bagga and Baldwin, 1998), Entity-based CEAF (CEAF e ) (Luo, 2005) and their average.”
Dynamic Entity Representations in Neural Language Models (2017, methodology): “It computes three commonly used evaluation measures MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF e (Luo, 2005).”
Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation (2014, methodology): “However, it turns out that the CEAF metric (Luo, 2005) was always intended to work seamlessly on predicted mentions, and so has been the case with the B 3 metric.”
Generating Referring Expressions in Context: The GREC Task Evaluation Challenges (2010, methodology): “We opted to use the following three: muc-6 [38], ceaf [23], and b-cubed [1], which seem to be the most widely accepted metrics.”
CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes (2012, background): “, 1995), ii) The mention based B-CUBED metric (Bagga and Baldwin, 1998) and iii) The entity based CEAF (Constrained Entity Aligned F-measure) metric (Luo, 2005).”
A Multi-Pass Sieve for Coreference Resolution (2010, neutral): “We refer the interested reader to (X. Luo, 2005; Finkel and Manning, 2008) for an analysis of these metrics.”
A Joint Model for Entity Analysis: Coreference, Typing, and Linking (2014, methodology): “Coreference results are reported using MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF e (Luo, 2005), as well as their average, the CoNLL metric, all computed from the reference implementation of the CoNLL scorer (Pradhan et al., 2014).”
Bootstrapping coreference resolution using word associations (2011, background): “We report recall, precision, and F 1 for MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF (Luo, 2005).”
Supervised Noun Phrase Coreference Research: The First Fifteen Years (2010, methodology): “To address these problems, two coreference scoring programs have been developed: B (Bagga and Baldwin, 1998) and CEAF (Luo, 2005).”
Entity-Centric Coreference Resolution with Model Stacking (2015, methodology): “The other metric used in our evaluation, Entity-based CEAFE (CEAFφ4) (Luo, 2005), was not used because it is expensive to compute.”
Coreference Resolution with World Knowledge (2011, methodology): “We employ two commonly-used scoring programs, B 3 (Bagga and Baldwin, 1998) and φ 3 -CEAF (Luo, 2005), both of which report results in terms of recall (R), precision (P), and F-measure (F) by comparing the gold-standard (i.e., key) partition, KP , against the system-generated (i.e., response)…”
Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution (2015, background): “01 (Pradhan et al., 2014; Luo et al., 2014), which scores based on 3 metrics, including MUC (Vilain et al., 1995), CEAF e (Luo, 2005), and B 3 (Bagga and Baldwin, 1998), as well as the CoNLL score, which is the arithmetic mean of the 3 metrics.”
RelaxCor Participation in CoNLL Shared Task on Coreference Resolution (2011, methodology): “Both parameters were empirically adjusted on the development set for the evaluation measure used in this shared task: the un-weighted average of MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998) and entity-based CEAF (Luo, 2005).”
Improving Coreference Resolution by Learning Entity-Level Distributed Representations (2016, methodology): “The improvement is largest in CEAFφ4 , which is encouraging because CEAFφ4 is the most recently proposed metric, designed to correct flaws in the other two (Luo, 2005).”
The GREC Named Entity Generation Challenge 2009: Overview and Evaluation Results (2009, methodology): “The numbers in the last three columns are the separate MUC , CEAF and B - CUBED F-Scores for each system, averaged over the two resolver tools (and rounded for reasons of space.”