Knowledge Enhanced Contextual Word Representations

By Matthew E. Peters, Mark Neumann, IV RobertL.Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. 2019

729 citations recorded by the source corpus.

Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update contextual word representations via a form of word-to-entity attention. In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that combines a small amount of entity linking supervision with a large amount of raw text. After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert) demonstrates improved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, entity typing, and word sense disambiguation. KnowBert’s runtime is comparable to BERT’s and it scales to large KBs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “We are particularly optimistic about generalizations of this work to (1) structured knowledge, which would result in a generalization of Peters et al. (2019) where we would also learn the decision of which entities are informative, (2) the multi-lingual setting, e.g., retrieving knowledge in a…”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “…into language models (Ahn et al., 2016; Yang et al., 2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wang et al., 2020 Memory Augmented Neural Networks Our entity memory layer is closely tied to memory-based…”
How Can We Know What Language Models Know? (2019, methodology): “Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhanced with entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “The recent rise of PLMs has sparked a discussion about whether these models can be used as Knowledge Bases (KBs) (Petroni et al., 2019, 2020; Davison et al., 2019; Peters et al., 2019; Jiang et al., 2020; Roberts et al., 2020).”
Modifying Memories in Transformer Models (2020, background): “Another line of work (Sun et al., 2019; Zhang et al., 2019; Peters et al., 2019) focuses on leveraging the readily available structured KBs to further complement the knowledge possessed by language models.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “The success of GPT-3 and recent studies on language models ’ ability in recalling factual knowledge (Petroni et al., 2019; Wang et al., 2020a; Liu et al., 2021b) has revealed the fact that Transformers can memorize.”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “KnowBERT (Peters et al., 2019) incorporates knowledge bases into BERT using Knowledge attention and recontextualization, where the knowledge comes from synset-synset and lemmalemma relationships in WordNet, and entity linking information in Wikipedia.”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, background): “Transformers in Vision Transformers [37] are the models that entirely rely on the self-attention mechanism to draw global dependencies between input and output, which has dominated in natural language modelling [10, 30, 2, 46, 29, 23].”
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (2019, methodology): “Compared with previous work (Zhang et al., 2019; Peters et al., 2019) that utilize external knowledge base to incorporate entity knowledge, our method is able to directly derive real-world knowledge from unstructured text.”
Pre-trained models for natural language processing: A survey (2020, methodology): “The external knowledge ranges from linguistic [94, 83, 136, 191], semantic [99], commonsense [52], factual [214, 136, 111, 202, 195], to domain-specific knowledge [58, 111].”
PTR: Prompt Tuning with Rules for Text Classification (2021, neutral): “, 2020), KNOWBERT (Peters et al., 2019), LUKE (Yamada et al.”
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2019, methodology): “Figure 3 shows that gains of E-BERT-concat over BERT, KnowBert and ERNIE in terms of mean Hits@k are especially big for k > 1 .”
CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, methodology): “Among these PLMs, GPT-3 (Brown et al., 2020), with 175 billion parameters and 570GB training data, has been the center of attention and proven to be effective in various few-shot (even zero-shot) NLP tasks.”
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA (2020, background): “ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019) are entity-enhanced versions of BERT.”
Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (2020, methodology): “For example, Peters et al. (2019) retrieve entities’ embeddings according to the similarity between a Transformer’s hidden states and pre-trained graph embeddings, then treat the retrieved embeddings as extra inputs to the next layer.”