Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

By Andrew S. Gordon, Zornitsa Kozareva, Melissa Roemmele. 2011

977 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “and commonsense reasoning (Roemmele et al., 2011; Mostafazadeh et al., 2016; Zhang et al., 2017).”
Language Models are Few-Shot Learners (2020, methodology): “pare to popular models such as BERT and RoBERTa in a more systematic way, we also evaluate GPT-3 on a standardized collection of datasets, the SuperGLUE benchmark [WPN+19] [WPN +19] [CLC 19] [DMST19] [RBG11] [KCR 18] [ZLL+18] [DGM06] [BHDD+06] [GMDD07] [BDD+09] [PCC18] [PHR+18]. GPT-3’s test-set per”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “• Sentence completion (COPA [Roemmele et al., 2011])”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “COPA (Choice of Plausible Alternatives, Roemmele et al., 2011) is a causal reasoning task in which a system is given a premise sentence and must determine either the cause or effect of the premise from two possible choices.”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, neutral): “edge, which cannot be explicitly decoded from the given text, is needed to solve the problem, as illustrated in the examples in Table1. Comparing with other commonsense reasoning tasks, such as COPA (Roemmele et al., 2011), Story Cloze Test (Mostafazadeh et al., 2016), Event2Mind (Rashkin et al.,201”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, methodology): “, 2018) and COPA (Gordon et al., 2012), the non-normalized loss can yield better results and thus is adopted.”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “, 2016) and has comparable performance on CoPA (Roemmele et al., 2011) and PiQA (Bisk et al.”
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories (2016, background): “There has also been renewed attention toward natural language comprehension and commonsense reasoning (Levesque, 2011; Roemmele et al., 2011; Bowman et al., 2015).”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, background): “The Choice of Plausible Alternatives (COPA) is a similarly important but small dataset consisting of 500 development and 500 test questions (Roemmele et al., 2011).”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “Table 2: Different prompt formulations of the COPA task (Roemmele et al., 2011) from SuperNaturalInstructions, PromptSource, FLAN and CrossFit.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, neutral): “…language inference 25.0 ✓ ✓ ✓ ✓ OpenBookQA (Mihaylov et al., 2018) Multiple-choice question answering 25.0 ✓ ✓ ✓ BoolQ (Clark et al., 2019) Multiple-choice question answering 50.0 ✓ ✓ ✓ COPA (Gordon et al., 2012) Sentence completion 50.0 ✓ ✓ CB (De Marneffe et al., 2019) Natural language…”
Improving and Simplifying Pattern Exploiting Training (2021, methodology): “For other tasks with multi-token labels (COPA (Roemmele et al., 2011), ReCoRD (Zhang et al., 2018)), P ET computes the probability of the classes as the sum of the log probabilities of the individual tokens.”
How many data points is a prompt worth? (2021, background): “These datasets comprise a variety of tasks, all in English, including entailment (MNLI, RTE (Dagan et al., 2005), CB (de Marneffe et al., 2019)), multiple choice question answering (BoolQ (Clark et al., 2019), MultiRC (Khashabi et al., 2018)), and commonsense reasoning (WSC (Levesque et al., 2012), ”