HellaSwag: Can a Machine Really Finish Your Sentence?
By Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi. 2019
4,547 citations recorded by the source corpus.
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as “A woman sits at a piano,” a machine must select the most likely followup: “She sets her fingers on the keys.” With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical ‘Goldilocks’ zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Subfields: Artificial Intelligence, Prompting & Reasoning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Hannaneh Hajishirzi
In Prompting & Reasoning
- Luke S. Zettlemoyer
- Yejin Choi
- Hannaneh Hajishirzi
- Mohit Bansal
- Denny Zhou
- Michael Lewis
- Peter Clark
- Tushar Khot
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Samuel R. Bowman
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Douwe Kiela
- Ashish Sabharwal
- Jonathan Berant
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “• Reading comprehension on RACE (Lai et al., 2017) • Common sense understanding on HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), Wino-grande (Sakaguchi et al., 2020), SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), and TruthfulQA (Lin et al., 2021).”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We consider eight standard common sense reasoning benchmarks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sak-aguchi et al., 2021), ARC easy and challenge (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018).”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We observe that only HellaSwag and MMLU-Humanities appear to have been boosted due to contamination in the training data, with the 70B model appearing to have gained a greater benefit than the 7B model, as one might expect.”
- Mistral 7B (2023, methodology): “We measure performance on a wide variety of tasks categorized as follow: • Commonsense Reasoning (0-shot): Hellaswag [28], Winogrande [21], PIQA [4], SIQA [22], OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] •…”
- Mixtral of Experts (2024, methodology): “We measure performance on a wide variety of tasks categorized as follow: • Commonsense Reasoning (0-shot): Hellaswag [32], Winogrande [26], PIQA [3], SIQA [27], OpenbookQA [22], ARC-Easy, ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] •…”
- Experience Grounds Language (2020, methodology): “Where Peters et al. (2018) introduced ELMo with ∼108 parameters, Transformer models (Vaswani et al., 2017) have continued to scale by orders of magnitude between papers (Devlin et al., 2019; Radford et al., 2019; Zellers et al., 2019b) to ∼1011 (Brown et al., 2020).”
- Adversarial Training for Large Neural Language Models (2020, methodology): “GPT (Zellers et al., 2019) 41.”
- PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “The dataset is further cleaned of basic artifacts using the AFLite algorithm introduced in (Sakaguchi et al. 2020; Sap et al. 2019) which is an improvement on adversarial fil-tering (Zellers et al. 2018; Zellers et al. 2019b).”
- Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “There has been a flurry of work in constructing datasets with an adversarial component, such as Swag (Zellers et al., 2018) and HellaSwag (Zellers et al., 2019), CODAH (Chen et al., 2019), Ad-versarial SQuAD (Jia and Liang, 2017), Lambada (Paperno et al., 2016) and others.”
- Defending Against Neural Fake News (2019, background): “Generating the first token after <startbody> results in high 13This matches findings on the HellaSwag dataset (Zellers et al., 2019b).”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “HellaSWAG HellaSWAG [76] is a commonsense reasoning dataset where a goal is given and the model is tasked with choosing the most likely follow-up actions.”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “In contrast, although fine-tuning of MoE models produces substantial benefits for Storycloze, BoolQ, SST-2, MNLI and some improvements on OpenBookQA, it results in worse performance for HellaSwag, PIQA, and Winogrande.”
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “, 2016), HellaSwag (Zellers et al., 2019), StoryCloze (Mostafazadeh et al.”
- Measuring Massive Multitask Language Understanding (2020, background): “While these benchmarks evaluate linguistic skills more than overall language understanding, an array of commonsense benchmarks have been proposed to measure basic reasoning and everyday knowledge (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2019).”
- Finetuned Language Models Are Zero-Shot Learners (2021, background): “HellaSwag (Zellers et al., 2019) tests for sentence completion that requires common sense, asking for the most probable ending given four contexts.”
- Shortcut learning in deep neural networks (2020, background): “object recognition, scene understanding, argument comprehension), it is important to bear in mind that a dataset is useful only for as long as it is a good proxy for the ability one is actually interested in [56, 76].”