Algorithms for Scoring Coreference Chains

By Amit Bagga, Breck Baldwin. 1998

Subfields: Evaluation & Benchmarks, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Information Extraction

Citations in the corpus

Toward Gender-Inclusive Coreference Resolution (2019, positive): “In terms of metrics, most intrinsic evaluation is carried out using metrics like MUC [Vilain et al. 1995], ACE [Mitchell et al. 2005], B 3 [Bagga and Baldwin 1998; Stoyanov et al. 2009], or CEAF [Luo 2005] (see also [Cai and Strube 2010] for additional discussion and variants).”
Deep Reinforcement Learning for Mention-Ranking Coreference Models (2016, methodology): “We use the B3 coreference metric for this reward (Bagga and Baldwin, 1998).”
Easy Victories and Uphill Battles in Coreference Resolution (2013, methodology): “For each experiment, we report MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF e (Luo, 2005), as well as their average, the CoNLL metric.”
Solving Hard Coreference Problems (2019, methodology): “Evaluation Metrics: When evaluating on the full datasets of ACE and OntoNotes , we use the widely recoginzed metrics MUC (Vilain et al., 1995), BCUB (Bagga and Baldwin, 1998), Entity-based CEAF (CEAF e ) (Luo, 2005) and their average.”
Dynamic Entity Representations in Neural Language Models (2017, methodology): “It computes three commonly used evaluation measures MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF e (Luo, 2005).”
Coreference Semantics from Web Features (2012, methodology): “We evaluated our work on both MUC (Vilain et al., 1995) and B 3 (Bagga and Baldwin, 1998).”
Coreference Resolution with World Knowledge (2011, methodology): “This is not the case when automatically extracted NPs are used, but the original description of B(3) does not specify how twinless NPs should be scored (Bagga and Baldwin, 1998).”
Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation (2014, methodology): “While the mathematical formulae for these metrics can be found in the original papers (Vilain et al., 1995; Bagga and Baldwin, 1998; Luo, 2005), many misunderstandings discussed in Section 2 are due to the fact that these papers lack an example showing how a metric is computed on predicted mentions.”
Entity-Centric Coreference Resolution with Model Stacking (2015, methodology): “For m, the performance metric determining the action costs, we use a linear combination of the B3 (Bagga and Baldwin, 1998) and MUC (Vilain et al.”
What Are You Talking About? Text-to-Image Coreference (2014, methodology): “Coreference metrics: We evaluate the predicted coreference clusters with MUC [32] and B(3) [1] metrics.”
Simple Coreference Resolution with Rich Syntactic and Semantic Features (2009, background): “• b 3 (Amit and Baldwin, 1998): For each mention, form the intersection between the predicted cluster and the true cluster for that mention.”
Event Detection and Co-reference with Minimal Supervision (2016, methodology): “For event co-reference, we compare all systems using standard F1 metrics: MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), Entity-based CEAF (CEAF e ) (Luo, 2005) and BLANC (Recasens and Hovy, 2011).”
Evaluating Entity Linking with Wikipedia (2013, methodology): “Systems are evaluated using clustering evaluation measures, such as The B 3 measure (Bagga and Baldwin, 1998a).”
Bootstrapping coreference resolution using word associations (2011, background): “We report recall, precision, and F 1 for MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF (Luo, 2005).”
A Joint Model for Entity Analysis: Coreference, Typing, and Linking (2014, methodology): “Coreference results are reported using MUC (Vilain et al., 1995), B 3 (Bagga and Baldwin, 1998), and CEAF e (Luo, 2005), as well as their average, the CoNLL metric, all computed from the reference implementation of the CoNLL scorer (Pradhan et al., 2014).”
Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric (2016, methodology): “Several evaluation metrics have been introduced for coreference resolution (Vilain et al., 1995; Bagga and Baldwin, 1998; Luo, 2005; Recasens and Hovy, 2011; Tuggener, 2014).”
Global joint models for coreference resolution and named entity classification (2009, background): “By focusing on the links, this metric has two main biases, which are now well-known (Bagga and Baldwin , 1998; Luo, 2005) but merit re-emphasis due its continued use as the sole evaluation measure.”
Generating Referring Expressions in Context: The GREC Task Evaluation Challenges (2010, methodology): “We opted to use the following three: muc-6 [38], ceaf [23], and b-cubed [1], which seem to be the most widely accepted metrics.”
Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, methodology): “Interms of metrics, most intrinsic evaluation is carried out using metrics like MUC (Vilain et al. 1995), ACE (Mitchell et al. 2005), B 3 (Bagga and Baldwin 1998; Stoyanov et al. 2009), or CEAF (Luo 2005) (see also (Cai and Strube 2010) for additional discussion and variants).”
Why discourse affects speakers’ choice of referring expressions (2015, methodology): “Interannotator agreement for CHILDES coreference annotation was computed using B(3) (Bagga and Baldwin, 1998): precision: 0.”