Planning, Executing, and Evaluating the Winograd Schema Challenge

By Leora Morgenstern, Ernest Davis, Charles L. Ortiz. 2016

66 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

WinoGrande (2019, methodology): “While improvements on some related datasets (particularly WSC, PDP, and DPR) might seem expected, the signiﬁcant improvement on COPA is not so.”
A Surprisingly Robust Trick for the Winograd Schema Challenge (2019, methodology): “This is particularly surprising, because previous work (Opitz and Frank, 2018) implies that generalizing to W SC 273 is hard.”
A Knowledge Hunting Framework for Common Sense Reasoning (2018, background): “These include Pronoun Disambiguation Prob-2 lems (more generalized, Winograd-like passages without the twist of a special word or twin) (Morgenstern et al., 2016), the Narrative cloze task (Taylor, 1953), or its more difﬁcult counterpart, the NarrativeQA Reading Comprehension Challenge (Koˇcisk`y…”
How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG (2018, unknown): “With the release of an increasing number of fine-grained inference tasks aimed at these abilities (Roemmele et al., 2011; Morgenstern et al., 2016; Wang et al., 2018; Rashkin et al., 2018; McCann et al., 2018), the issue of experimental validity in CSR will also become even more important.”
An Adversarial Winograd Schema Challenge at Scale (2019, methodology): “PDP (Morgenstern et al., 2016) PDP (Pronoun Disambiguation Problems) dataset is closely related to the original WSC, and used in the 2016”
Precise Task Formalization Matters in Winograd Schema Evaluations (2020, methodology): “The SuperGLUE version of the task recasts examples from WSC273 and the Pronoun Disambiguation Problems dataset (Morgenstern et al., 2016) into 554 training and 104 validation binary classification problems.”
Commonsense Reasoning for Natural Language Understanding: A Survey of Benchmarks, Resources, and Approaches (2019, methodology): “The classic coreference resolution benchmark is the Winograd Schema Challenge (WSC), inspired by Winograd (1972), originally proposed by Levesque (2011), later developed by Levesque, Davis, and Morgenstern (2012), and actualized by Morgenstern and Ortiz (2015) and Morgenstern et al. (2016).”
Combing Context and Commonsense Knowledge Through Neural Networks for Solving Winograd Schema Problems (2016, methodology): “…disambiguation problems (PDP) are complex coreference resolution problems, which are taken directly or modiﬁed from examples found in literature, biographies, au-tobiographies, essays, news analyses, and news stories; they may need some manual processing (Morgenstern, Davis, and Ortiz Jr 2016).”
On the Evaluation of Common-Sense Reasoning in Natural Language Understanding (2018, background): “With the release of an increasing number of ﬁne-grained inference tasks aimed at these abilities [7, 5, 9], we hope to set a precedent for future work and emphasize the importance of analysis for future empirical studies.”
CorefQA: Coreference Resolution as Query-based Span Prediction (2019, background): “Coreference resolution is a fundamental problem in natural language processing and is considered as a good test of machine intelligence (Morgenstern et al., 2016).”