An Adversarial Winograd Schema Challenge at Scale

By Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi. 2019

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “• Reading comprehension on RACE (Lai et al., 2017) • Common sense understanding on HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), Wino-grande (Sakaguchi et al., 2020), SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), and TruthfulQA (Lin et al., 2021).”
LLaMA: Open and Efficient Foundation Language Models (2023, background): “We consider eight standard common sense reasoning benchmarks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sak-aguchi et al., 2021), ARC easy and challenge (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018).”
Mistral 7B (2023, methodology): “We measure performance on a wide variety of tasks categorized as follow: • Commonsense Reasoning (0-shot): Hellaswag [28], Winogrande [21], PIQA [4], SIQA [22], OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] •…”
Mixtral of Experts (2024, methodology): “We measure performance on a wide variety of tasks categorized as follow: • Commonsense Reasoning (0-shot): Hellaswag [32], Winogrande [26], PIQA [3], SIQA [27], OpenbookQA [22], ARC-Easy, ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] •…”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, positive): “…MCTest (Richard-son et al., 2013), RACE (Lai et al., 2017), OpenBookQA/OBQA (Mihaylov et al., 2018), ARC (Clark et al., 2018, 2016), QASC (Khot et al., 2019), CommonsenseQA/CQA (Talmor et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), and Winogrande (Sakaguchi et al., 2020).”
TTTTTackling WinoGrande Schemas (2020, neutral): “We applied the T5 sequence-to-sequence model [5] to tackle the AI2 WinoGrande Challenge [6] by decomposing each example into two input text strings, each containing a hypothesis, and using the probabilities assigned to the “entailment” token as a score of the hypothesis.”
PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “The AFLite algorithm performs a systematic data bias reduction: it discards instances whose given feature representations are collectively highly indicative of the target label.”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “Winogrande Winogrande [58] is a dataset that seeks to expand the Winograd Schema Challenge in both scale and difﬁculty.”
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, methodology): “To gauge the model’s common sense reasoning we evaluate it on the Winogrande Schema Challenge (Sakaguchi et al., 2020).”
OPT: Open Pre-trained Transformer Language Models (2022, background): “We evaluate our model on 16 standard NLP tasks utilized in the literature: HellaSwag (Zellers et al., 2019), StoryCloze (Mostafazadeh et al., 2016), PIQA (Bisk et al., 2020), ARC Easy and Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), WinoGrad (Levesque et al., 2011), Wino-Grand”
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, neutral): “To mitigate the issue of repetitive writing, we also provide workers with an inspiration prompt, that crowdworkers may optionally use as a starting point in their writing, this is similar to the data collection procedure for WinoGrande (Sakaguchi et al., 2019).”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, neutral): “Tasks Type Random small core main ext HellaSwag (Zellers et al., 2019) Sentence completion 25.0 ✓ ✓ ✓ ✓ LAMBADA (Paperno et al., 2016) Sentence completion 0.0 ✓ ✓ ✓ Winogrande (Sakaguchi et al., 2021) Coreference resolution 50.0 ✓ ✓ ✓ ✓ PIQA (Bisk et al., 2020) Multiple-choice question answering 50.”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, methodology): “• Core-knowledge benchmarks, including MMLU [16], HellaSwag [41], ARC [8], WinoGrande [31], HumanEval [6], GSM-8K [9], and AGIEval [42], evaluate the core capabilities of pre-trained LLMs using zero-shot and few-shot benchmark sets.”
Few-Shot Text Generation with Natural Language Instructions (2020, methodology): “The idea to rephrase tasks as cloze questions is commonly used to probe the knowledge contained within masked language models (e.g., Petroni et al., 2019; Wang et al., 2019; Talmor et al., 2020; Schick and Schütze, 2020; Ettinger, 2020; Kassner and Schütze, 2020; Sakaguchi et al., 2020).”
Natural Adversarial Examples (2019, background): “Many recently introduced NLP datasets use adversarial filtration to create “adversarial datasets” by sieving examples solved with simple spurious cues [44, 4, 56, 13, 6, 25].”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, positive): “From each collection, we use the datasets that include English training data: – GLUE: CoLA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), QQP, STS-B (Cer et al., 2017), MNLI (Williams et al., 2018), QNLI (Converted from Rajpurkar et al. (2016), RTE (Dagan et al.”
Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations (2021, methodology): “…that the WG-dev accuracy in Table 4 is much higher than the corresponding WG-test accuracy in Table 3, because WG-dev is sampled from the training dataset of WinoGrande, whereas WG-test is the original WinoGrande development set, which is ﬁltered to increase its difﬁculty (Sakaguchi et al., 2020).”