SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

By Rowan Zellers, Yonatan Bisk, Roy Schwartz, Yejin Choi. 2018

795 citations recorded by the source corpus.

Given a partial description like “she opened the hood of the car,” humans can reason about the situation and anticipate what might come next (”then, she examined the engine”). In this paper, we introduce the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning. We present SWAG, a new dataset with 113k multiple choice questions about a rich spectrum of grounded situations. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, we propose Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data. To account for the aggressive adversarial filtering, we use state-of-the-art language models to massively oversample a diverse set of potential counterfactuals. Empirical results demonstrate that while humans can solve the resulting inference problems with high accuracy (88%), various competitive models struggle on our task. We provide comprehensive analysis that indicates significant opportunities for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “The Situations With Adversarial Generations (SWAG) dataset contains 113k sentence-pair completion examples that evaluate grounded commonsense inference (Zellers et al., 2018).”
Experience Grounds Language (2020, methodology): “This disconnect is driven home by studies of bias (Gururangan et al., 2018; Glockner et al., 2018) and techniques like Adversarial Filtering (Zellers et al., 2018, 2019b), which elucidate the biases models exploit in lieu of understanding.”
Adversarial Training for Large Neural Language Models (2020, methodology): “• SWAG .”
PIQA: Reasoning about Physical Commonsense in Natural Language (2019, methodology): “The dataset is further cleaned of basic artifacts using the AFLite algorithm introduced in (Sakaguchi et al. 2020; Sap et al. 2019) which is an improvement on adversarial ﬁl-tering (Zellers et al. 2018; Zellers et al. 2019b).”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “There has been a ﬂurry of work in constructing datasets with an adversarial component, such as Swag (Zellers et al., 2018) and HellaSwag (Zellers et al., 2019), CODAH (Chen et al., 2019), Ad-versarial SQuAD (Jia and Liang, 2017), Lambada (Paperno et al., 2016) and others.”
WinoGrande (2019, methodology): “…of the data and we collect their predictions 6 A F L ITE is designed for ﬁltering instances so that the resulting dataset is less biased, whereas the original AF algorithm (Zellers et al. 2018) is designed for “generating and modifying” individual instances, such as by creating better distractors.”
Defending Against Neural Fake News (2019, background): “One additional possibility is the use of Adversarial Filtering (Zellers et al., 2018; 2019b) to oversample and then select a subset of generations.”
Story Ending Prediction by Transferable BERT (2019, background): “Next Action Prediction SWAG (Situations With Adversarial Generations) contains 113k sentence-pair completion examples that evaluate commonsense inference [ Zellers et al. , 2018 ] .”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “Similarly, given the similarity of COPA to SWAG (Zellers et al., 2018), we first fine-tune BERT on SWAG.”
HellaSwag: Can a Machine Really Finish Your Sentence? (2019, methodology): “Zellers et al. (2018) used a two-layer LSTM for generation, with shallow stylistic adversarial ﬁl-ters.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “a recent trend in creating datasets with adversarial baselines in the loop (Paperno et al., 2016; Minervini and Riedel, 2018; Zellers et al., 2018; Zhang et al., 2019; Zellers et al., 2019).”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “(2) SWAG uniﬁes commonsense reasoning and natural language inference.”
Reasoning Like Program Executors (2022, methodology): “…Skills The literature focuses on rea- 112 soning skills including numerical reasoning (Dua 113 et al., 2019), multi-hop reasoning (Yang et al., 114 2018), reasoning in hybrid context (Chen et al., 115 2020b; Zhu et al., 2021) and logical reasoning (Liu 116 et al., 2020; Yu et al., 2020).”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, methodology): “However, it has been quickly realized that models trained on large amounts of unlabeled data (Devlin et al., 2018) capture well this type of information and performance on SWAG is already at human level.”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, background): “‚ SWAG is a large-scale adversarial dataset for the task of grounded commonsense inference, which unifies natural language inference and physically grounded reasoning [35].”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “Others currently-supported benchmarks include SWAG (Zellers et al. (2018)), RACE (Lai et al. (2017)) and ARC (Clark et al. (2018)).”
oLMpics-On What Language Model Pre-training Captures (2019, methodology): “We adapt the architecture to the multi-choice setup using the procedure proposed by Zellers et al. (2018).”