Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers

By Anupam Guha, Mohit Iyyer, Danny Bouman, Jordan L. Boyd-Graber. 2015

30 citations recorded by the source corpus.

Coreference is a core nlp problem. However, newswire data, the primary source of existing coreference data, lack the richness necessary to truly solve coreference. We present a new domain with denser references—quiz bowl questions—that is challenging and enjoyable to humans, and we use the quiz bowl community to develop a new coreference dataset, together with an annotation framework that can tag any text data with coreferences and named entities. We also successfully integrate active learning into this annotation pipeline to collect documents maximally useful to coreference models. State-of-the-art coreference systems underperform a simple classifier on our new dataset, motivating non-newswire data for future coreference research.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Toward Gender-Inclusive Coreference Resolution (2019, methodology): “Considering the approaches used in dominant papers, the active learning ∨ ∧ algorithms used are not statistically unbiased [Guha et al. 2015; Laws et al. 2012; Miller et al. 2012; Ng and Cardie 2003; Sachan et al. 2015].”
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning (2019, methodology): “Guha et al. (2015) present the limitations of annotating coreference in newswire texts alone, and like us, built a non-newswire coreference resolution dataset focusing on Quiz Bowl questions.”
Toward Gender-Inclusive Coreference Resolution: An Analysis of Gender and Bias Throughout the Machine Learning Lifecycle* (2021, methodology): “used in dominant papers, the active learning algorithms used are not statistically unbiased (Ng and Cardie 2003; Laws, Heimerl, and Schütze 2012; Miller, Dligach, and Savova 2012; Sachan, Hovy, and Xing 2015; Guha et al. 2015).”
Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions (2018, background): “Many questions have complicated coreference patterns (Guha et al., 2015), require reasoning across multiple types of knowledge, or involve wordplay.”