Evaluation of named entity coreference

By Oshin Agarwal, Sanjay Subramanian, Ani Nenkova, Dan Roth. 2019

16 citations recorded by the source corpus.

In many NLP applications like search and information extraction for named entities, it is necessary to find all the mentions of a named entity, some of which appear as pronouns (she, his, etc.) or nominals (the professor, the German chancellor, etc.). It is therefore important that coreference resolution systems are able to link these different types of mentions to the correct entity name. We evaluate state-of-the-art coreference resolution systems for the task of resolving all mentions to named entities. Our analysis reveals that standard coreference metrics do not reflect adequately the requirements in this task: they do not penalize systems for not identifying any mentions by name to an entity and they reward systems even if systems find correctly mentions to the same entity but fail to link these to a proper name (she–the student–no name). We introduce new metrics for evaluating named entity coreference that address these discrepancies and show that for the comparisons of competitive systems, standard coreference evaluations could give misleading results for this task. We are, however, able to confirm that the state-of-the art system according to traditional evaluations also performs vastly better than other systems on the named entity coreference task.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Toward Gender-Inclusive Coreference Resolution (2019, methodology): “As observed by Agarwal et al. [2019], most of these metrics are rather insensitive to arguably large errors, like the inability to link pronouns to names; to address this, they introduce a new metric to focus specifically on this named entity coreference task.”
Neural Data Augmentation via Example Extrapolation (2021, background): “…“The [1 sun] sets in the [0 north]” • “The [1 sun] sets in the [0 south]” Ensuring that models make decisions via language understanding rather than memorizing facts or entities has been argued for named entity recognition (Agarwal et al., 2020) and coreference resolution (Agarwal et al., 2019).”