HellaSwag: Can a Machine Really Finish Your Sentence?

By Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi. 2019

4,547 citations recorded by the source corpus.

Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as “A woman sits at a piano,” a machine must select the most likely followup: “She sets her fingers on the keys.” With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical ‘Goldilocks’ zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.

Subfields: Artificial Intelligence, Prompting & Reasoning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Yejin Choi
  6. Graham Neubig
  7. Noah A. Smith
  8. Hannaneh Hajishirzi

In Prompting & Reasoning

  1. Luke S. Zettlemoyer
  2. Yejin Choi
  3. Hannaneh Hajishirzi
  4. Mohit Bansal
  5. Denny Zhou
  6. Michael Lewis
  7. Peter Clark
  8. Tushar Khot

In Evaluation & Benchmarks

  1. Mohit Bansal
  2. Yejin Choi
  3. Samuel R. Bowman
  4. Luke S. Zettlemoyer
  5. Hannaneh Hajishirzi
  6. Douwe Kiela
  7. Ashish Sabharwal
  8. Jonathan Berant

Citations in the corpus