CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge

By Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, Greg Durrett. 2021

107 citations recorded by the source corpus.

Most benchmark datasets targeting commonsense reasoning focus on everyday scenarios: physical knowledge like knowing that you could fill a cup under a waterfall [Talmor et al., 2019], social knowledge like bumping into someone is awkward [Sap et al., 2019], and other generic situations. However, there is a rich space of commonsense inferences anchored to knowledge about specific entities: for example, deciding the truthfulness of a claim"Harry Potter can teach classes on how to fly on a broomstick."Can models learn to combine entity knowledge with commonsense reasoning in this fashion? We introduce CREAK, a testbed for commonsense reasoning about entity knowledge, bridging fact-checking about entities (Harry Potter is a wizard and is skilled at riding a broomstick) with commonsense inferences (if you're good at a skill you can teach others how to do it). Our dataset consists of 13k human-authored English claims about entities that are either true or false, in addition to a small contrast set. Crowdworkers can easily come up with these statements and human performance on the dataset is high (high 90s); we argue that models should be able to blend entity knowledge and commonsense reasoning to do well here. In our experiments, we focus on the closed-book setting and observe that a baseline model finetuned on existing fact verification benchmark struggles on CREAK. Training a model on CREAK improves accuracy by a substantial margin, but still falls short of human performance. Our benchmark provides a unique probe into natural language understanding models, testing both its ability to retrieve facts (e.g., who teaches at the University of Chicago?) and unstated commonsense knowledge (e.g., butlers do not yell at guests).

Subfields: Artificial Intelligence, Management Science and Operations Research, Prompting & Reasoning, Evaluation & Benchmarks, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Prompting & Reasoning

In Evaluation & Benchmarks

In Question Answering & Reading Comprehension

Citations in the corpus

Graph Retrieval-Augmented Generation: A Survey (2024, unknown): “This iterative process helps both components refine their performance in a coordinated manner.”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, methodology): “To compare with ODIE which covers relatively new entities originated in 2017 at the earliest, we use a set of P OPULAR entities ranked by article contributor numbers and incoming links from prior work (Onoe et al., 2021; Geva et al., 2021).”
Many-shot Jailbreaking (2024, background): “…observe that in-context learning follows predictable power laws on the following safety-unrelated datasets: LogiQA (Liu et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sakaguchi et al., 2019), TriviaQA (Joshi et al., 2017), Common-senseQA (Talmor et al., 2019), CREAK (Onoe et al., 2021).”
Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (2023, background): “Moreover, in order to examine ToG on more generic tasks, we also prepare one open-domain QA dataset: WebQuestions (Berant et al., 2013); two slot filling datasets: T-REx (ElSahar et al., 2018) and Zero-Shot RE (Petroni et al., 2021); and one fact-checking dataset: Creak (Onoe et al., 2021).”
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (2023, unknown): “And fourth, unlike previous work, we localize LLM truth representations to a small number of hidden states above certain tokens.”
A Unified Encoder-Decoder Framework with Entity Memory (2022, background): “Many entity-intensive NLP tasks require models obtain entity knowledge to generate informative outputs, such as answering factual questions (Kwiatkowski et al., 2019), explaining claims (Onoe et al., 2021), or making informative conversations (Dinan et al., 2019).”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, methodology): “In Com2Sense test set and CREAK contrast set, each question is paired with its complimentary counterpart, of which the surface form is similar but the answer should be the opposite (e.g. “Barack Obama has daughters.” vs “Barack Obama has no daughter.””