Evaluation Metrics For End-to-End Coreference Resolution Systems

By Jie Cai, Michael Strube. 2010

80 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Greg Durrett
2. Christopher D. Manning
3. Kevin Clark
4. Daniel Klein
5. Alessandro Moschitti
6. Dan Roth
7. Sameer Pradhan
8. Mohit Bansal

Citations in the corpus

• Toward Gender-Inclusive Coreference Resolution (2019, positive): “In terms of metrics, most intrinsic evaluation is carried out using metrics like MUC [Vilain et al. 1995], ACE [Mitchell et al. 2005], B 3 [Bagga and Baldwin 1998; Stoyanov et al. 2009], or CEAF [Luo 2005] (see also [Cai and Strube 2010] for additional discussion and variants).”
• Easy Victories and Uphill Battles in Coreference Resolution (2013, methodology): “Note that this version of the scorer implements a modified version of B, described in Cai and Strube (2010), that was used for the CoNLL shared tasks.”
• CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes (2011, methodology): “The modifications suggested by Cai and Strube (2010) were incorporated in the scorer.”
• CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes (2012, methodology): “The modifications suggested by Cai and Strube (2010) have been incorporated in the scorer.”
• Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation (2014, methodology): “Given space constraints we use an example worked out in Cai and Strube (2010).”
• Towards Robust Linguistic Analysis using OntoNotes (2013, methodology): “This relates to the mis-implementation of the method proposed by (Cai and Strube, 2010) for scoring predicted mentions.”
• RelaxCor Participation in CoNLL Shared Task on Coreference Resolution (2011, methodology): “Regarding the evaluation, the scorer uses the modification of (Cai and Strube, 2010), unprecedented so far, and the corpora was published very recently and there are no published results yet to use as reference.”
• Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, methodology): “Interms of metrics, most intrinsic evaluation is carried out using metrics like MUC (Vilain et al. 1995), ACE (Mitchell et al. 2005), B 3 (Bagga and Baldwin 1998; Stoyanov et al. 2009), or CEAF (Luo 2005) (see also (Cai and Strube 2010) for additional discussion and variants).”