CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
By Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant. 2019
2,559 citations recorded by the source corpus.
When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Percy Liang
- Noah A. Smith
- Graham Neubig
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Mohit Bansal
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
In Natural Language Processing
- Mohit Bansal
- Yejin Choi
- Luke S. Zettlemoyer
- Graham Neubig
- Jianfeng Gao
- Hannaneh Hajishirzi
- Percy Liang
- Noah A. Smith
In Question Answering & Reading Comprehension
- Ashish Sabharwal
- Tushar Khot
- Peter Clark
- Hannaneh Hajishirzi
- Kyle Richardson
- Jonathan Berant
- Yejin Choi
- Xiang Ren
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, background): “, 2017), as well as the commonsense reasoning datasets CommonsenseQA (Talmor et al., 2019) and StrategyQA (Geva et al.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We report the average of PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019a), WinoGrande (Sakaguchi et al., 2021), ARC easy and challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and CommonsenseQA (Talmor et al., 2018).”
- Mistral 7B (2023, background): “…• Commonsense Reasoning (0-shot): Hellaswag [28], Winogrande [21], PIQA [4], SIQA [22], OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] • Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10]…”
- Mixtral of Experts (2024, background): “…• Commonsense Reasoning (0-shot): Hellaswag [32], Winogrande [26], PIQA [3], SIQA [27], OpenbookQA [22], ARC-Easy, ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot)…”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, neutral): “We find that with the same prompts, chain-of-thought prompting improves performance across all three models (LaMDA, GPT-3, and PaLM) for all datasets except CSQA and StrategyQA for GPT-3 (Table 1, Table 4, Table 5).”
- UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, positive): “…MCTest (Richard-son et al., 2013), RACE (Lai et al., 2017), OpenBookQA/OBQA (Mihaylov et al., 2018), ARC (Clark et al., 2018, 2016), QASC (Khot et al., 2019), CommonsenseQA/CQA (Talmor et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), and Winogrande (Sakaguchi et al., 2020).”
- WinoGrande (2019, methodology): “guistics and the structural constraints of twin sentences.(Trichelair et al., 2018; Talmor et al., 2018) One novelty of our work is that we demonstrate a method to collect WSC problems at scale through crowdsourcing.”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Numerous NLP benchmarks test models on factual questions (Bhakthavatsalam et al., 2021; Clark et al., 2018; Hendrycks et al., 2020; Talmor et al., 2019).”
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “For these tasks we used the CommonsenseQA (Talmor et al., 2019) benchmark for commonsense reasoning, StrategyQA (Geva et al.”
- Reasoning Like Program Executors (2022, background): “Our work concentrates on improving the above reasoning skills, leaving the other reasoning abilities such as commonsense reasoning (Zellers et al., 2018; Talmor et al., 2019; Bhagavatula et al., 2020) for future work.”
- Training Verifiers to Solve Math Word Problems (2021, background): “Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
- oLMpics-On What Language Model Pre-training Captures (2019, methodology): “To relax this we use in two tasks the standard setup for answering multi-choice questions with pre-trained LMs (Talmor et al., 2019; Mihaylov et al., 2018).”
- Large Language Models are Zero-Shot Reasoners (2022, methodology): “Table 5 shows the performance of Few-shot-CoT when using examples from different datasets: CommonsenseQA to AQUA-RAT and CommonsenseQA to MultiArith.”
- UL2: Unifying Language Learning Paradigms (2022, positive): “, 2019), CommonsenseQA (Talmor et al., 2018), CommonsenseQA2 (Talmor et al.”