Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric

By Nafise Sadat Moosavi, Michael Strube. 2016

166 citations recorded by the source corpus.

Interpretability and discriminative power are the two most basic requirements for an evaluation metric. In this paper, we report the mention identiﬁcation effect in the B 3 , CEAF , and BLANC coreference evaluation metrics that makes it impossible to interpret their results properly. The only metric which is insensitive to this ﬂaw is MUC , which, however, is known to be the least discriminative metric. It is a known fact that none of the current metrics are reliable. The common practice for ranking coreference resolvers is to use the average of three different metrics. However, one cannot expect to obtain a reliable score by averaging three unreliable metrics. We propose LEA, a Link-based Entity-Aware evaluation metric that is designed to overcome the shortcomings of the current evaluation metrics. LEA is available as branch LEA-scorer in the reference implementation of the ofﬁcial CoNLL scorer.

Subfields: Artificial Intelligence, Management Science and Operations Research, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Toward Gender-Inclusive Coreference Resolution (2019, methodology): “Table 5 (left) reports results according the standard coreference resolution evaluation metric LEA [Moosavi and Strube 2016].”
Evaluation of named entity coreference (2019, background): “Prior work (Moosavi and Strube, 2016) discussed shortcoming of these metrics and introduced the improved link entity aware (LEA) score.”
Adapting Coreference Resolution for Processing Violent Death Narratives (2021, methodology): “We saw E2E-Aug further improved E2E-FT by 5% on LEA F1 with the 30 LGBT narratives in VDN’s test set9.”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, background): “, 2016), (ii) B 3 : a single-mention based metric which computes the macro F1 of all entity mentions, and (iii) CEAF φ 4 : a similarity metric based on the assumption that the coreference map should be one-to-one.”