How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG

By Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung. 2018

44 citations recorded by the source corpus.

Recent studies have significantly improved the state-of-the-art on common-sense reasoning (CSR) benchmarks like the Winograd Schema Challenge (WSC) and SWAG. The question we ask in this paper is whether improved performance on these benchmarks represents genuine progress towards common-sense-enabled systems. We make case studies of both benchmarks and design protocols that clarify and qualify the results of previous work by analyzing threats to the validity of previous experimental designs. Our protocols account for several properties prevalent in common-sense benchmarks including size limitations, structural regularities, and variable instance difficulty.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Richard S. Zemel
2. Jörn-Henrik Jacobsen
3. Matthias Bethge
4. Wieland Brendel
5. Kyunghyun Cho
6. Yoshua Bengio
7. Graham Neubig
8. Mohit Bansal

In Electrical and Electronic Engineering

1. Matthias Bethge
2. Phil Blunsom
3. Richard S. Zemel
4. Jörn-Henrik Jacobsen

In Natural Language Processing

1. Richard S. Zemel
2. Phil Blunsom
3. A. Kuncoro
4. Aida Nematzadeh
5. Cyprien de Masson d'Autume
6. Matthias Bethge
7. Claudio Michaelis
8. Goran Nenadic

In Evaluation & Benchmarks

1. Wieland Brendel
2. Matthias Bethge
3. Richard S. Zemel
4. A. Kuncoro
5. Riza Theresa Batista-Navarro
6. Cyprien de Masson d'Autume
7. Phil Blunsom
8. Moin Nabi

Citations in the corpus

• Shortcut learning in deep neural networks (2020, background): “Initially designed to overcome shortcut opportunities caused by the open-ended nature of the Turing test, this common-sense reasoning benchmark was scrutinised after modern language models started to perform suspiciously well—and it indeed contained more shortcut opportunities than originally envisi”
• Contrastive Self-Supervised Learning for Commonsense Reasoning (2020, background): “In this case, the supervised fine-tuning does not seem to be enough to solve the task, and can only learn to “artificially” resolve the pronoun based on superficial cues such as dataset and language biases (Trichelair et al., 2018; Saba, 2018; Trichelair et al., 2019; Emami et al., 2019; Kavumba et ”
• Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets (2020, unknown): “Consistency In addition to using accuracy to measure models’ performance, recent research pays attention to consistency, which provides another perspective to probe models’ competence in the real world (Trichelair et al., 2018; Zhou et al., 2019; Gardner et al., 2020).”
• A Systematic Investigation of Commonsense Understanding in Large Language Models (2021, background): “Trichelair et al. (2019) study the Answer-only Baseline on one commonsense benchmark, SWAG (Zellers et al., 2019a).”
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, methodology): “…Baselines (Sugawara et al., 2020) Stress-test (Dua et al., 2019a) Manual Analyses (Yatskar, 2019) SWAG Partial Baselines (Sugawara et al., 2020; Trichelair et al., 2019) Adversarial Evaluation (Zellers et al., 2019; Zellers et al., 2018) CNN Manual Analyses (Chen et al., 2016) Partial…”