Commonsense Knowledge Mining from Pretrained Models

By Joshua Feldman, Joe Davison, Alexander M. Rush. 2019

355 citations recorded by the source corpus.

Inferring commonsense knowledge is a key challenge in machine learning. Due to the sparsity of training data, previous work has shown that supervised methods for commonsense knowledge mining underperform when evaluated on novel data. In this work, we develop a method for generating commonsense knowledge using a large, pre-trained bidirectional language model. By transforming relational triples into masked sentences, we can use this model to rank a triple’s validity by the estimated pointwise mutual information between the two entities. Since we do not update the weights of the bidirectional model, our approach is not biased by the coverage of any one commonsense knowledge base. Though we do worse on a held-out test set than models explicitly trained on a corresponding training set, our approach outperforms these methods when mining commonsense knowledge from new sources, suggesting that our unsupervised technique generalizes better than current supervised approaches.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “Recently, the same idea has been applied to unsupervised text classification (Puri and Catanzaro, 2019) and commonsense knowledge mining (Davison et al., 2019).”
Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “…we borrow components from discrete prompt search methods, such as prompt generation (Gao et al., 2021; Ben-David et al., 2021), prompt scoring (Davison et al., 2019) and prompt paraphrasing (Jiang et al., 2020; Yuan et al., 2021) to optimize instructions by searching directly in the natural…”
Making Pre-trained Language Models Better Few-shot Learners (2021, background): “Finally, there is a large body of work on prompting for mining knowledge from pre-trained models (Trinh and Le, 2018; Petroni et al., 2019; Davison et al., 2019; Talmor et al., 2020, inter alia).”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “Enabling LMs to perform zero-shot learning by providing task descriptions was proposed by Radford et al. (2019) and has been applied to text classiﬁ-cation (Puri and Catanzaro, 2019), commonsense knowledge mining (Davison et al., 2019) and ar-gumentative relation classiﬁcation (Opitz, 2019).”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “[19] investigate the task of knowledge base completion and design a template for an input (head-relation-tail triple) using LMs.”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “One of the remarkable observations in recent machine learning research is that large language models (LLMs) can not only generate fluent textual descriptions, but also appear to have rich internalized knowledge about the world [3, 4, 5, 6, 7].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): ““Dante was born in ⟨PLACE⟩”) can be extracted (Petroni et al., 2019; Jiang et al., 2020; Davison et al., 2019; Talmor et al., 2020; Roberts et al., 2020).”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “We include as a baseline an unsupervised ﬁltration metric inspired by (Davison et al., 2019): they propose a model estimate of PMI to score mined commonsense triples.”
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (2021, background): “…et al., 2019; Peters et al., 2018; Raffel et al., 2020; Liu et al., 2019; Yang et al., 2019; Radford et al., 2019) have shown remarkable performance when conditioned with an appropriate textual context (Petroni et al., 2019, 2020; Jiang et al., 2020; Shin et al., 2020; Davison et al., 2019).”
GPT Understands, Too (2021, background): “, 2019b), syntactic (Hewitt & Manning, 2019), commonsense (Davison et al., 2019) and even world knowledge (Petroni et al.”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “The recent rise of PLMs has sparked a discussion about whether these models can be used as Knowledge Bases (KBs) (Petroni et al., 2019, 2020; Davison et al., 2019; Peters et al., 2019; Jiang et al., 2020; Roberts et al., 2020).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “Davison et al. (2019) suggest that it generalizes better to unseen data.”
Pre-Trained Models: Past, Present and Future (2021, background): “Then, to extract commonsense from PTMs, Davison et al. (2019) propose to first transform relational triples into masked sentences and then rank these sentences according to the mutual information given by PTMs.”
Unsupervised Commonsense Question Answering with Self-Talk (2020, methodology): “, 2019) and BERT (Devlin et al., 2019) capture various types of world knowledge. Petroni et al. (2019) showed that such LMs can be used in a KB completion task over ConceptNet and Wikidata (Vrandečić and Krötzsch, 2014) by converting KB relations into natural language templates and querying the LM f”
Pre-trained models for natural language processing: A survey (2020, background): “cient queries. Despite the surprising ﬁndings of LAMA, it has also been questioned by subsequent work [141, 82]. Similarly, several studies induce relational knowledge [15] and commonsense knowledge [32] from BERT for downstream tasks. 4 Extensions of PTMs 4.1 Knowledge-Enriched PTMs PTMs usually le”
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020, background): “ PET with a manually deﬁned mapping. 2 Related Work Reformulating problems as language modeling tasks has been explored in fully unsupervised settings (Radford et al., 2019; Puri and Catanzaro, 2019; Davison et al., 2019), in few-shot scenarios with limited amounts of training data (Opitz, 2019; Shw”
PTR: Prompt Tuning with Rules for Text Classification (2021, background): “ries of research work on knowledge probing (Trinh and Le, 2018; Petroni et al., 2019; Davison et al., 2019), language prompts have been proven to effectively stimulate knowledge from PLMs.”
Entailment as Few-Shot Learner (2021, background): “…GPT works (Radford et al., 2019; Brown et al., 2020) proposed to add a task description (prompt) and annotated examples as demonstration to enable few-shot learning, which has been commonly applied to classification (Puri & Catanzaro, 2019), QA, commonsense knowledge mining (Davison et al., 2019).”