COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

By Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, Yejin Choi. 2020

465 citations recorded by the source corpus.

Recent years have brought about a renewed interest in commonsense representation and reasoning in the field of natural language understanding. The development of new commonsense knowledge graphs (CSKG) has been central to these advances as their diverse facts can be used and referenced by machine learning models for tackling new and challenging tasks. At the same time, there remain questions about the quality and coverage of these resources due to the massive scale required to comprehensively encompass general commonsense knowledge. In this work, we posit that manually constructed CSKGs will never achieve the coverage necessary to be applicable in all situations encountered by NLP agents. Therefore, we propose a new evaluation framework for testing the utility of KGs based on how effectively implicit knowledge representations can be learned from them. With this new goal, we propose Atomic 2020, a new CSKG of general-purpose commonsense knowledge containing knowledge that is not readily available in pretrained language models. We evaluate its properties in comparison with other leading CSKGs, performing the first large-scale pairwise study of commonsense knowledge resources. Next, we show that Atomic 2020 is better suited for training knowledge models that can generate accurate, representative knowledge for new, unseen entities and events. Finally, through human evaluation, we show that the few-shot performance of GPT-3 (175B parameters), while impressive, remains ~12 absolute points lower than a BART-based knowledge model trained on Atomic 2020 despite using over 430x fewer parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Flexible Generation of Natural Language Deductions (2021, background): “These relationships can be formalized into a monotonicity calculus (Icard et al., 2017) and past work has grounded lexical inference tasks into such a formalism (Angeli et al., 2016; Hu et al., 2020).”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “As a result, symbolic commonsense knowledge graphs (Speer et al., 2017; Sap et al., 2019; Hwang et al., 2020) and their corresponding neural representations (Bosselut et al.”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…could be jointly or sequentially trained on static text and common-sense knowledge bases or socially grounded or interactional text (Bosselut et al., 2019; Hwang et al., 2021), ﬁrst trained on static text and then enhanced for commonsense knowledge via reinforcement learning (Zhou et al., 2021).”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, background): “…(Zellers et al., 2018, 2019), cultural, social, and physical commonsense understanding (Lin et al., 2018; Sap et al., 2019a,b; Bisk et al., 2020; Hwang et al., 2020; Forbes et al., 2020), pronoun disambiguation (Sakaguchi et al., 2020; Zhang et al., 2020), abductive commonsense reasoning…”
Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences (2020, background): “Previous efforts to model intentions underlying social actions and their consequences (Rashkin et al., 2018; Hwang et al., 2020) largely regard actions in isolation, without taking into account their broader situational context or norm conformity.”
CIDER: Commonsense Inference for Dialogue Explanation and Reasoning (2021, background): “However, it has been proved that declarative knowledge is still valuable, especially implicit relationships that are hardly acquired by the state of the art models (Hwang et al., 2020).”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “ATOMIC [72], [73] and ASER [74] focus on the causal effects between events, which can be used for commonsense reasoning.”
Graph Retrieval-Augmented Generation: A Survey (2024, unknown): “Formally, each node representation h ( 𝑙 − 1 ) in the 𝑙 -th layer is updated by aggregating the information from neighboring nodes and edges: , e ( 𝑙 − 1 )”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, methodology): “Hwang et al. (2021) use the same training framework with more powerful LMs like BART (Lewis et al.”
DREAM: Uncovering Mental Models behind Language Models (2021, background): “Differing from just generating a “likely consequence” (found in the COMET dataset (Hwang et al., 2020)), this setup is intended to generate consequences that are contrastive (in terms of producing good or bad outcome), to assist in situational QA tasks.”
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering (2022, background): “Certain approaches learn to access implicit knowledge encoded in LMs (Bosselut et al., 2019; Petroni et al., 2019; Hwang et al., 2021) by training on structured KG data, and then use the LM to generate local KGs that can be used for QA (Wang et al., 2020; Bosselut et al., 2021).”
Mapping Language Models to Grounded Conceptual Spaces (2022, background): “We analyse where models fail and see that the smallest models often produce random, unrelated outputs, but the errors made by the largest model are quite intuitive, for example, predicting colours very close in space to the true grounding.”
Knowledge-Augmented Methods for Natural Language Processing (2023, background): “present the recent works on developing commonsense knowledge resources [16, 50], which provide us with event-centric, large-scale, neural-symbolic, semi-structured ways to access and model commonsense knowledge.”
Natural Language Processing with Commonsense Knowledge: A Survey (2021, methodology): “Liu et al. [98] proposed a novel approach that uses the COMET-BART model [22] to extract implicit commonsense knowledge from the dialogue context.”
SafeText: A Benchmark for Exploring Physical Safety in Language Models (2022, methodology): “For each relation template (Causes, isBefore, and isAfter), we calculate the perplexity of generating the tail entity with COMET-GPT2.”
Think Before You Speak: Explicitly Generating Implicit Commonsense Knowledge for Response Generation (2021, background): “We plan to explore other knowledge resources such as ATOMIC2020 (Hwang et al., 2021) in the future.”
APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation (2024, methodology): “Following the emergence of the COMET [6] model pretrained on the commonsense corpus, some studies are beginning to focus on integrating external common knowledge into empathetic response tasks.”