COMET: Commonsense Transformers for Automatic Knowledge Graph Construction
By Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, Yejin Choi. 2019
1,043 citations recorded by the source corpus.
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017). Contrary to many conventional KBs that store knowledge with canonical templates, commonsense KBs only store loosely structured open-text descriptions of knowledge. We posit that an important step toward automatic commonsense completion is the development of generative models of commonsense knowledge, and propose COMmonsEnse Transformers (COMET) that learn to generate rich and diverse commonsense descriptions in natural language. Despite the challenges of commonsense modeling, our investigation reveals promising results when implicit knowledge from deep pre-trained language models is transferred to generate explicit knowledge in commonsense knowledge graphs. Empirical results demonstrate that COMET is able to generate novel knowledge that humans rate as high quality, with up to 77.5% (ATOMIC) and 91.7% (ConceptNet) precision at top 1, which approaches human performance for these resources. Our findings suggest that using generative commonsense models for automatic commonsense KB completion could soon be a plausible alternative to extractive methods.
Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Knowledge Graphs & Representation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yejin Choi
- Percy Liang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Noah A. Smith
- Hannaneh Hajishirzi
- Graham Neubig
In Molecular Biology
- Christopher D. Manning
- Percy Liang
- Xiang Ren
- Daniel Klein
- Phil Blunsom
- Mohit Bansal
- Noah A. Smith
- Iryna Gurevych
In Natural Language Processing
- Yejin Choi
- Mohit Bansal
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Noah A. Smith
- Graham Neubig
- Percy Liang
- Jianfeng Gao
In Knowledge Graphs & Representation
- Yejin Choi
- Ronan Le Bras
- Xiang Ren
- Jure Leskovec
- Mohit Bansal
- Adyasha Maharana
- Michihiro Yasunaga
- Chandra Bhagavatula
Citations in the corpus
- PIQA: Reasoning about Physical Commonsense in Natural Language (2019, neutral): “Within NLP, in addition to large scale models, there has also been progress on reasoning about cause and effect effects/implications within these models (Bosselut et al. 2019), extracting knowledge from them (Petroni et al.”
- HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “, 2019) and commonsense inference (Bosselut et al., 2019).”
- UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “N ET (Speer, Chin, and Havasi 2017) and A TOMIC (Sap et al. 2019a), to create additional knowledge graph generation tasks (Bosselut et al. 2019).”
- Calibrate Before Use: Improving Few-Shot Performance of Language Models (2021, methodology): “Asking LMs to complete natural language prompts is also used as a method to “probe” LMs, e.g., analyzing their factual (Petroni et al., 2019; Jiang et al., 2020b; Shin et al., 2020) or commonsense knowledge (Bosselut et al., 2019).”
- Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “Evaluation follows past work (Hwang et al., 2021; Bosselut et al., 2019; Sap et al., 2019) testing the ability of models to do knowledge base completion, i.e. generating inferences for test events, specifically from the A TOMIC 2020 test set.”
- Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “Promising results on commonsense reasoning suggest a possible hybrid approach: LLMs could be jointly or sequentially trained on static text and commonsense knowledge bases or socially grounded or interactional text (Bosselut et al., 2019; Hwang et al., 2021), first trained on static text and then”
- HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “, 2019), commonsense inference (Bosselut et al., 2019), and summarization (Lewis et al.”
- Mass-Editing Memory in a Transformer (2022, background): “Pioneering hand-curated efforts (Lenat, 1995; Miller, 1995) have been followed by web-powered knowledge graphs (Auer et al., 2007; Bollacker et al., 2007; Suchanek et al., 2007; Havasi et al., 2007; Carlson et al., 2010; Dong et al., 2014; Vrandečić & Krötzsch, 2014; Bosselut et al., 2019) that extr”
- Social Bias Frames: Reasoning about Social and Power Implications of Language (2019, methodology): “Satisfying these conditions, we use the OpenAI-GPT transformer networks (Vaswani et al., 2017; Radford et al., 2018, 2019) as a basis for our experiments, given their recent successes at classifcation, commonsense generation, and conditional generation (Bosselut et al., 2019; Keskar et al., 2019).”
- Commonsense Knowledge Mining from Pretrained Models (2019, background): “Bosselut et al. (2019) and Trinh and Le (2019) use unidirectional language models for CKBC, but their approach requires a supervised training step.”
- Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (2019, background): “Bosselut et al. (2019) investigate PLMs’ common sense knowledge, but do not”
- Pre-Trained Models: Past, Present and Future (2021, background): “Since aligning entities and relations to raw text is often troublesome and can introduce noise in data pre-processing, another line of works (Bosselut et al., 2019; Guan et al., 2020; Chen et al., 2020d) can directly convert structural knowledge into the serialized text and let models learn knowledg”
- Unsupervised Commonsense Question Answering with Self-Talk (2020, methodology): “COMeT (Bosselut et al., 2019) is a knowledge base construction model trained on the ATOMIC resource (Sap et al., 2019a) which consists of everyday situations along with multiple commonsense dimensions such as their causes, effects, pre- and post-conditions, etc.”
- Inducing Relational Knowledge from BERT (2019, methodology): “Finally, Bosselut et al. (2019) propose an automatic knowledge graph construction method based on transformer language models.”
- GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, methodology): “We use the 710K training split introduced in Bosselut et al. (2019) to randomly sample 100K tuples as the seed sub-graphs for generation of synthetic data dataset for Social IQA.”
- K-BERT: Enabling Language Representation with Knowledge Graph (2019, methodology): “COMET (Bosselut et al. 2019) employed the triples in KG as corpus to train GPT (Radford et al.”
- Latent Hatred: A Benchmark for Understanding Implicit Hate Speech (2021, methodology): “and commonsense reasoning (Sap et al., 2020; Forbes et al., 2020; Sharma et al., 2020; Bosselut et al., 2019), we fine-tune Open-AI’s GPT (Radford et al.”