Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation

By Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard H. Hovy, Vincent Ng, Michael Strube. 2014

212 citations recorded by the source corpus.

The definitions of two coreference scoring metrics— B3 and CEAF—are underspecified with respect to predicted, as opposed to key (or gold) mentions. Several variations have been proposed that manipulate either, or both, the key and predicted mentions in order to get a one-to-one mapping. On the other hand, the metric BLANC was, until recently, limited to scoring partitions of key mentions. In this paper, we (i) argue that mention manipulation for scoring predicted mentions is unnecessary, and potentially harmful as it could produce unintuitive results; (ii) illustrate the application of all these measures to scoring predicted mentions; (iii) make available an open-source, thoroughly-tested reference implementation of the main coreference evaluation measures; and (iv) rescore the results of the CoNLL-2011/2012 shared task systems with this implementation. This will help the community accurately measure and compare new end-to-end coreference resolution algorithms.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods (2018, background): “01 (Pradhan et al., 2014) and compute the average (Avg) and absolute difference (Diff) between pro-stereotyped and anti-stereotyped conditions in WinoBias.”
Dynamic Entity Representations in Neural Language Models (2017, methodology): “For coreference resolution evaluation, we employ the CoNLL scorer (Pradhan et al., 2014).”
Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric (2016, methodology): “In this section, we use the example from Pradhan et al. (2014) to show the process of computing the LEA scores.”
Using Coreference Links to Improve Spanish-to-English Machine Translation (2017, methodology): “These metrics are implemented in the scorer for CoNLL 2012 (Pradhan et al., 2014) and the SemEval 2013 one (Màrquez et al., 2013).”
Robustness in Coreference Resolution (2020, methodology): “The CoNLL-2012 shared task dataset (Pradhan et al., 2012) is the largest available corpus that is annotated with coreference information. After the introduction of this dataset, it became the most prominent corpus in the coreference literature. Henceforth, we refer to this dataset as CoNLL or CoNLL-”
Evaluation of named entity coreference (2019, methodology): “Shared tasks on coreference (at CoNLL-2011 and 2012 (Pradhan et al., 2014) ) use the average of three F1 scores as their ofﬁcial evaluation: MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998) and CEAFE (Luo, 2005).”
Learning Global Features for Coreference Resolution (2016, methodology): “Scoring uses the ofﬁcial CoNLL 2012 script (Pradhan et al., 2014; Luo et al., 2014).”
Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution (2015, background): “01 (Pradhan et al., 2014; Luo et al., 2014), which scores based on 3 metrics, including MUC (Vilain et al., 1995), CEAF e (Luo, 2005), and B 3 (Bagga and Baldwin, 1998), as well as the CoNLL score, which is the arithmetic mean of the 3 metrics.”
A Joint Model for Entity Analysis: Coreference, Typing, and Linking (2014, methodology): “Coreference results are reported using MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF e (Luo, 2005), as well as their average, the CoNLL metric, all computed from the reference implementation of the CoNLL scorer (Pradhan et al., 2014).”
An Adversarial Winograd Schema Challenge at Scale (2019, methodology): “Although the size is larger than Winogender, WinoBias is evaluated by F-scores (i.e., detecting the span) as a coreference resolution task (Pradhan et al., 2014) instead of the binary choice accuracy as in WSC. SuperGLUE (Wang et al., 2019) SuperGLUE contains multiple datasets for universal…”