Inducing Relational Knowledge from BERT

By Zied Bouraoui, José Camacho-Collados, Steven Schockaert. 2019

184 citations recorded by the source corpus.

One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships. Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks. However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings. To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model. Starting from a few seed instances of a given relation, we first use a large text corpus to find sentences that are likely to express this relation. We then use a subset of these extracted sentences as templates. Finally, we fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “For relation extraction, Bouraoui et al. (2020) automatically identify sentences that express given relations and then use these sentences as patterns.”
Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, methodology): “…by the recent ﬁnding that pretrained language models encode various types of factual knowledge and commonsense knowledge in their parameters [15–17], we employ UniLM [18]—a uniﬁed pre-trained language generation model that achieved state-of-the-art performance on many text generation…”
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (2021, background): “…from its training corpora (Rogers et al., 2020), including factual knowledge (Petroni et al., 2019; Jiang et al., 2020), common sense (Bisk et al., 2019), reasoning (Talmor et al., 2020; Brown et al., 2020), summarization (Radford et al., 2019), and even arithmetic (Bouraoui et al., 2020).”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “However, to retrieve BERT’s knowledge we need good template sentences, and there is work on their automatic extraction and augmentation (Bouraoui et al., 2019; Jiang et al.)”
Eliciting Knowledge from Language Models Using Automatically Generated Prompts (2020, background): “…commonsense reasoning (Trinh and Le, 2018; Kwon et al., 2019; Shwartz et al., 2020), question answering (Lewis et al., 2019), fact recall (Petroni et al., 2019; Jiang et al., 2020; Bouraoui et al., 2019), summarization (Radford et al., 2019), and other supervised tasks (Brown et al., 2020).”
Pre-Trained Models: Past, Present and Future (2021, background): “In addition to linguistic knowledge, PTMs also learn rich world knowledge from pre-training, mainly including commonsense knowledge and factual knowledge (Zhou et al., 2020b; Bouraoui et al., 2020).”
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models (2020, background): “[26,5] and [42] demonstrate that BERT has surprisingly strong ability to recall factual relational knowledge and perform commonsense reasoning.”
Pre-trained models for natural language processing: A survey (2020, background): “Similarly, several studies induce relational knowledge [15] and commonsense knowledge [31] from BERT for downstream tasks.”
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020, methodology): “odel has been studied by Jiang et al. (2019). This is also closely related to approaches for extracting patterns in relation extraction (Brin, 1999; Agichtein and Gravano, 2000; Batista et al., 2015; Bouraoui et al., 2020). 3 Pattern-Exploiting Training We review Pattern-Exploiting Training (PET) as”
Birds Have Four Legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models (2020, positive): “For example, (Petroni et al., 2019; Davison et al., 2019; Bouraoui et al., 2020) ﬁnd that it is feasible to use PTLMs as a commonsense knowledge base.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “They do so by either using prompts and letting the LM fill in the blanks, which assumes that the LM is a static knowledge source (Petroni et al., 2019; Jiang et al., 2020; Poerner et al., 2019; Bouraoui et al., 2020), or fine-tuning the LM on a set of question-answer pairs to directly generate answe”
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021, unknown): “…aspects, such as negation (Kassner and Sch¨utze, 2020), easy to guess names (Poerner et al., 2020), integrating adapters (Wang et al., 2020) or ﬁnding alternatives to a “ﬁll-in-the-blank” approach with single-token answers (Bouraoui et al., 2020; Heinzerling and Inui, 2020; Jiang et al., 2020b).”
O N M EMORY IN H UMAN AND A RTIFICIAL L ANGUAGE P ROCESSING S YSTEMS (2020, background): “It has been shown that relatively low-level knowledge ( e.g. , word similarity)—which would be stored in a semantic memory—can be extracted from these models in an unsupervised way (Petroni et al., 2019; Bouraoui et al., 2020).”
BERTese: Learning to Speak to BERT (2021, background): “ods for extracting that knowledge (Petroni et al., 2019, 2020; Jiang et al., 2020; Bouraoui et al., 2020).”
CoLAKE: Contextualized Language and Knowledge Embedding (2020, methodology): “…help to denoise distantly annotated samples of relation extraction (Craven and Kumlien, 1999; Mintz et al., 2009), (b) CoLAKE can be used to measure the quality of graph-to-text templates (Davison et al., 2019; Bouraoui et al., 2020) due to its capability of preserving the original graph structure.”
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020, methodology): “We speculate that data-driven methods (Jiang et al., 2020; Bouraoui et al., 2019) can be designed to mitigate (2), but will exacerbate (1).”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, methodology): “…approaches convert KB triples to sentences using relation templates in order to construct task-speciﬁc training datasets for improved performance in, e.g., story generation (Guan et al., 2020), commonsense QA (Ye et al., 2020; Ma et al., 2021), and relation classiﬁcation (Bouraoui et al., 2020).”
Towards a Universal Continuous Knowledge Base (2020, background): “2 Knowledge Bases from Pre-trained Models Recently, a number of works have been studied on what does the pre-trained language model learns (Petroni et al., 2019; Bouraoui et al., 2020; Rogers et al., 2020; Wang et al., 2020).”
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021, methodology): “We therefore also investigate a probing setup (Alain and Bengio, 2017; Bouraoui et al., 2020), in which a representation induced by a pretrained model is provided to a second probing model which is trained to predict attributes of interest.”