WinoGrande

By Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi. 2019

344 citations recorded by the source corpus.

Commonsense reasoning remains a major challenge in AI, and yet, recent progresses on benchmarks may seem to suggest otherwise. In particular, the recent neural language models have reported above 90% accuracy on the Winograd Schema Challenge (WSC), a commonsense benchmark originally designed to be unsolvable for statistical models that rely simply on word associations. This raises an important question---whether these models have truly acquired robust commonsense capabilities or they rely on spurious biases in the dataset that lead to an overestimation of the true capabilities of machine commonsense. To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) large-scale crowdsourcing, followed by (2) systematic bias reduction using a novel AFLITE algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. Our experiments demonstrate that state-of-the-art models achieve considerably lower accuracy (59.4%-79.1%) on WINOGRANDE compared to humans (94%), confirming that the high performance on the original WSC was inflated by spurious biases in the dataset. Furthermore, we report new state-of-the-art results on five related benchmarks with emphasis on their dual implications. On the one hand, they demonstrate the effectiveness of WINOGRANDE when used as a resource for transfer learning. On the other hand, the high performance on all these benchmarks suggests the extent to which spurious biases are prevalent in all such datasets, which motivates further research on algorithmic bias reduction.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We report the average of PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019a), WinoGrande (Sakaguchi et al., 2021), ARC easy and challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and CommonsenseQA (Talmor et al., 2018).”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “, 2012) and variants thereof as cloze tasks to investigate commonsense reasoning abilities of language models (Trinh and Le, 2018; Sakaguchi et al., 2019).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “Following Brown et al. (2020), we use k=50 few-shot examples for WinoGrande, k=70 for StoryCloze and k=100 for OpenBookQA.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “Open-Domain Question Answering: TriviaQA (Joshi et al., 2017), Natural Questions (NQS) (Kwiatkowski et al., 2019), Web Questions (WebQS) (Berant et al., 2013) Cloze and Completion Tasks: LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), StoryCloze (Mostafazadeh et al., 2016) Winograd”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “FLAN with the best dev template outperforms zero-shot GPT-3 on Winogrande (Sakaguchi et al., 2020).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Due to space constraints a representative subset of the results are shown here, with the rest in Appendix E. Natural Language Tasks We evaluate our model on a diverse collection of standard language model evaluation datasets: ANLI (Nie et al., 2020), ARC (Clark et al., 2018), HeadQA (English) (Vilar”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, background): “Although numerous studies have been done to quantify biases in various Natural language processing (NLP) tasks such as coreference resolution and word embeddings [2, 7, 29, 30], there has been limited work addressing biases in open-ended natural language generation.”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “We analyzed crowdsourcing templates of 6 datasets: CosmosQA (Huang et al., 2019), DROP (Dua et al., 2019), MC-TACO (Zhou et al., 2019), QASC (Khot et al., 2020), Quoref (Dasigi et al., 2019), and Winogrande (Sakaguchi et al., 2020).”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, methodology): “Our work joins a line of previously introduced benchmarks targeting commonsense reasoning 365 [12, 13, 29, 15] as well as probing for other reasoning abilities [32, 33, 34, 35, 36, 37, 38].”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Since in MIP, we have included the Winogrande (Sakaguchi et al., 2021) and SuperGLUE WSC (Wang et al., 2019), here we test on Winogender (Rudinger et al., 2018) and Winograd273 (Levesque et al., 2012).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, neutral): “…al. 2016 wiki_qa cls/other Yang et al. 2015 wiki_split cg/other Botha et al. 2018 wikisql cg/other Zhong et al. 2017 wino_grande qa/multiple-choice qa Sakaguchi et al. 2020 wiqa qa/multiple-choice qa Tandon et al. 2019 xsum cg/summarization Narayan et al. 2018 yahoo_answers_topics cls/topic (link)…”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, methodology): “The A F L ITE algorithm proposed by Sakaguchi et al. [2020] provides a way to systematically identify dataset instances that are easily gamed by a model, but in ways that are not easily detected by humans.”
Reframing Instructional Prompts to GPTk’s Language (2021, methodology): “Example Table 1 (row 2) illustrates the WinoGrande (Sakaguchi et al., 2020) sample generation task where the raw instructions contain several requisites (do’s and don’ts) that are hard for models to follow.”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, methodology): “, 2019), WINOGRANDE (Sakaguchi et al., 2020), HELLASWAG (Zellers et al.”