Explanations for CommonsenseQA: New Dataset and Models

By Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, Dinesh Garg. 2021

166 citations recorded by the source corpus.

CommonsenseQA (CQA) (Talmor et al., 2019) dataset was recently released to advance the research on common-sense question answering (QA) task. Whereas the prior work has mostly focused on proposing QA models for this dataset, our aim is to retrieve as well as generate explanation for a given (question, correct answer choice, incorrect answer choices) tuple from this dataset. Our explanation definition is based on certain desiderata, and translates an explanation into a set of positive and negative common-sense properties (aka facts) which not only explain the correct answer choice but also refute the incorrect ones. We human-annotate a first-of-its-kind dataset (called ECQA) of positive and negative properties, as well as free-flow explanations, for 11K QA pairs taken from the CQA dataset. We propose a latent representation based property retrieval model as well as a GPT-2 based property generation model with a novel two step fine-tuning procedure. We also propose a free-flow explanation generation model. Extensive experiments show that our retrieval model beats BM25 baseline by a relative gain of 100% in F_1 score, property generation model achieves a respectable F_1 score of 36.4, and free-flow generation model achieves a similarity score of 61.9, where last two scores are based on a human correlated semantic similarity metric.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, neutral): “For Common-senseQA, 3 / 3 preferred the GPT-3 explanation to the CoS-E explanation, and 2 / 3 to the ECQA one. which we expect the model to generate an explanation.”
Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “We exclude C O S-E (Rajani et al., 2019) as it is too noisy to be useful for modeling and evaluating self-rationalization (Narang et al., 2020), but we do not support using C O S-E in the future, especially since ECQA is introduced.”
Making Large Language Models Better Reasoners with Alignment (2023, unknown): “Specifically, the task accuracy and the assessment accuracy exhibit a strong positive correlation, with Pearson Correlation Coefficients of 0.93 and 0.98 at GSM8K and ECQA, respectively.”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “11 [100] commonsense QA crowd 10,962 ECQA [2] commonsense QA crowd 10,962 SEN-MAKING [124] commonsense validation students + authors 2,021 CHANGEMYVIEW [10] argument persuasiveness crowd 37,718 WINOWHY [144] pronoun coreference resolution crowd 273 (5) SBIC [111] social bias inference crowd 48,923 (”
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, unknown): “These are: (1) StrategyQA (Geva et al., 2021), (2) CommonsenseQA (CSQA; (Aggarwal et al., 2021; Talmor et al., 2019)), (3) GSM8K (Cobbe et al., 2021), (4) AQuA (Ling et al., 2017), (5) MATH (Hendrycks et al., 2021), (6) Date Understanding (BIG-bench collaboration, 2023), and (7) ANLI (Nie et al.,…”
REV: Information-Theoretic Evaluation of Free-Text Rationales (2022, methodology): “For automatic metrics, the evaluation models of R EV and the proxy models of LAS and RQ are trained on the ECQA training set with gold labels and human-annotated rationales (§3.2).”
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022, background): “There has been signiﬁcant progress made in recent years on both curating natural language explanation datasets (Camburu et al., 2018; Rajani et al., 2019; Brahman et al., 2021; Aggarwal et al., 2021, inter alia ) as well as generating them (Rajani et al., 2019; Shwartz et al., 2020).”