Differentiable Open-Ended Commonsense Reasoning

By Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, William W. Cohen. 2020

45 citations recorded by the source corpus.

Current commonsense reasoning research focuses on developing models that use commonsense knowledge to answer multiple-choice questions. However, systems designed to answer multiple-choice questions may not be useful in applications that do not provide a small list of candidate answers to choose from. As a step towards making commonsense reasoning research more realistic, we propose to study open-ended commonsense reasoning (OpenCSR) — the task of answering a commonsense question without any pre-defined choices — using as a resource only a corpus of commonsense facts written in natural language. OpenCSR is challenging due to a large decision space, and because many questions require implicit multi-hop reasoning. As an approach to OpenCSR, we propose DrFact, an efficient Differentiable model for multi-hop Reasoning over knowledge Facts. To evaluate OpenCSR methods, we adapt several popular commonsense reasoning benchmarks, and collect multiple new answers for each test question via crowd-sourcing. Experiments show that DrFact outperforms strong baseline methods by a large margin.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “For sparse methods, this goal is typically achieved with an inverted index—a space-efﬁcient technique entertaining the support of multiple robust libraries such as Pyserini (Lin et al., 2021b).”
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge (2021, background): “Finally, some QA datasets have been adapted for evaluating differentiable commonsense reasoning models [Lin et al., 2021], but these benchmarks still test very different knowledge from ours.”
Unsupervised Cross-Task Generalization via Retrieval Augmentation (2022, methodology): “The proposed ReCross pipeline is inspired by open-ended QA methods such as DPR (Karpukhin et al., 2020), DrFact (Lin et al., 2021), and RAG (Lewis et al., 2020b).”
Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “We then introduce the popular datasets for evaluating the commonsense reasoning methods that span three main categories: 1) multiple-choice QA (e.g., CommonsenseQA (Tal-mor et al., 2019), SocialIQA (Sap et al., 2019), PhysicalIQA (Bisk et al., 2020), RiddleSense (Lin et al., 2021b)), 2) open-ended Q”
Think Before You Speak: Using Self-talk to Generate Implicit Commonsense Knowledge for Response Generation (2021, background): “Commonsense Reasoning The majority of recent CSR benchmarks (Zellers et al., 2018; Tal-mor et al., 2019; Bisk et al., 2020; Sap et al., 2019; Lin et al., 2021c,a, 2020) test a model’s ability to choose the correct option given a context and a question.”
A Systematic Investigation of Commonsense Understanding in Large Language Models (2021, background): “…where a model needs to choose the right answer from a list of candidates including the correct choice and a few incorrect ones; (b) the generative evaluation (Boratko et al., 2020; Lin et al., 2020, 2021) which requires a model to generate an answer given a question and some additional context.”