The Winograd Schema Challenge

By Hector J. Levesque, Ernest Davis, Leora Morgenstern. 2012

50 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Language Models are Few-Shot Learners (2020, neutral): “ement with model capacity. This is shown in Figure3.4in the case of few-shot results, and scaling for all three settings is shown in AppendixH. 3.4 Winograd-Style Tasks The Winograd Schemas Challenge [LDM12] is a classical task in NLP that involves determining which word a pronoun refers to, when th”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “Some recent work also interprets the Winograd Schema Challenge (Levesque et al., 2012) and variants thereof as cloze tasks to investigate commonsense reasoning abilities of language models (Trinh and Le, 2018; Sakaguchi et al.”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, neutral): “, 2009) and WNLI (Levesque et al., 2012).”
A Simple Method for Commonsense Reasoning (2018, methodology): “This task consists of 273 questions and is designed to work against techniques such as traditional linguistic restrictions, common heuristics or simple statistical test over text corpora ("Google-proof ") [1].”
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, background): “, 2013)), linguistic acceptability (CoLA), and Winograd Schema (WNLI (Levesque et al., 2012)).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, neutral): “, 2007), and WNLI (Levesque et al., 2011).”
Shortcut learning in deep neural networks (2020, background): “ed time and resources, one might want to focus on challenging test cases. As models evolve, generalisation benchmarks will need to evolve as well, which is exempliﬁed by the Winograd Schema Challenge [103]. Initially designed to overcome shortcut opportunities caused by the open-ended nature of the ”
Toward Gender-Inclusive Coreference Resolution (2019, methodology): “(2018) evaluates coreference systems for evidence of occupational stereotyping, by constructing Winograd-esque (Levesque et al., 2012) test examples.”
How many data points is a prompt worth? (2021, background): “, 2018)), and commonsense reasoning (WSC (Levesque et al., 2012), COPA (Roemmele et al.”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, methodology): “In addition to the OntoNotes coreference examples, we include an extra “challenge” coreference dataset based on the Winograd schema (Levesque et al., 2012).”
ERNIE: Enhanced Language Representation with Informative Entities (2019, background): “, 2018) is a collection of diverse natural language understanding tasks (Warstadt et al., 2018; Socher et al., 2013; Dolan and Brockett, 2005; Agirre et al., 2007; Williams et al., 2018; Rajpurkar et al., 2016; Dagan et al., 2006; Levesque et al., 2011), which is the main benchmark used in Devlin et”
ERNIE 2.0: A Continual Pre-training Framework for Language Understanding (2019, methodology): “inference tasks. RTE: Recognizing Textual Entailment (RTE)[21] is a corpus similar to MNLI and is usually used for Natural Language Inference task. WNLI: Winograd Natural Language Inference (WNLI)[22] is a corpus that captures the coreference information between two paragraphs. QQP: Quora Questio”
GPT-3: Its Nature, Scope, Limits, and Consequences (2020, background): “3 See for example the Winograd Schema Challenge (Levesque et al. 2012).”
Probabilistic Reasoning via Deep Learning: Neural Association Models (2016, background): “The main contributions of this paper can be summarized as follows: • We propose neural association models for probabilistic reasoning based on deep neural networks, which are general enough to deal with various reasoning problems over symbolic events. f B ( L+ 1)”
Improving Question Answering by Commonsense-Based Pre-Training (2018, unknown): “Commonsense reasoning is a major challenge for question answering (Levesque, Davis, and Morgenstern 2011; Clark et al. 2018; Ostermann et al. 2018; Boratko et al. 2018).”
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections (2020, background): “…QQP (Quora Question Pairs) [Iyer et al., 2017], Semantic Textual Similarity Benchmark (STSB) [Cer et al., 2017], MNLI (Multi-Genre Natural Language Inference) Williams et al. [2018], QNLI [Rajpurkar et al., 2016], RTE [Dagan et al., 2005], Winograd Schema Challenge WNLI [Levesque et al., 2012].”
Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework (2017, methodology): “The DPR dataset is a collection of such problems and their solutions, collected by Rahman and Ng (2012) as a step towards solving the Winograd Schema Challenge (Hector et al., 2012).”
CERT: Contrastive Self-supervised Learning for Language Understanding (2020, background): “• WNLI In the Winograd Schema Challenge (Levesque et al., 2012), the task is to read a sentence with a pronoun and select the reference of that pronoun from a list of choices.”
On our best behaviour (2014, background): “6This section is drawn mainly from [8].”