E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

By Nina Poerner, Ulli Waltinger, Hinrich Schütze. 2019

168 citations recorded by the source corpus.

We present a novel way of injecting factual knowledge about entities into the pretrained BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT’s native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to ERNIE (Zhang et al., 2019) and KnowBert (Peters et al., 2019), but it requires no expensive further pre-training of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E-BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that someone with an Italian-sounding name speaks Italian), and that E-BERT mitigates this problem.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

True Few-Shot Learning with Language Models (2021, methodology): “We only use examples from the LAMA-UnHelpfulNames subset [LAMA-UHN; 57] which ﬁlters out easy-to-guess examples (e.g., “The Apple Watch was created by _” with the answer Apple ).”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “robustly encoded in the model, compared to mere ‘guesses’, which may arise from some heuristic or spurious correlations with certain patterns (Poerner et al., 2020).”
Towards Continual Knowledge Learning of Language Models (2021, background): “We also remove instances where the answer overlapped with the subject following Poerner et al. (2019) since the answers for these instances can be inferred from the cloze statement itself.”
Meta-learning via Language Model In-context Tuning (2021, methodology): “In our experiments, we use the TREx-UHN portion of LAMA (Poerner et al., 2020), which consists of (subject, relation, object) triples from Wikidata.”
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020, background): “They do so by either using prompts and letting the LM fill in the blanks, which assumes that the LM is a static knowledge source (Petroni et al., 2019; Jiang et al., 2020; Poerner et al., 2019; Bouraoui et al., 2020), or fine-tuning the LM on a set of question-answer pairs to directly generate answe”
A Review on Language Models as Knowledge Bases (2022, background): “, 2021), dealing with uncommon facts (Poerner et al., 2020; Jiang et al., 2020a) and counterfactual reasoning (Feng et al.”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, methodology): “In this situation, we can expect a “good" model to perform an educated (that is, non-random) guess, e.g., by relying on the association between the name “Enrico Fermi" and Italy when predicting Rome as the birthplace of the physicist (Poerner et al., 2020).”
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2020, background): “However, at the same time, LMs are obviously not omnipotent, and still fail to provide appropriate answers in many cases, such as when dealing with uncommon facts (Poerner et al., 2019; Jiang et al., 2020a) or complex reasoning (Talmor et al., 2019a).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Poerner et al. (2020) [186] introduced LAMA-UHN which is a collection of triples from the LAMA probing benchmark which are difficult to guess.”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, methodology): “…links—mid/early fusion (§ 4.3) Entity embedding retrieval (Peters et al., 2019; Févry et al., 2020) Treating entities as tokens (Yamada et al., 2020; Poerner et al., 2020) not only further our understanding of the effects of word-level supervision on LM knowledge acquisition, but will also…”
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021, unknown): “LAMA-UHN is a subset of triples that are hard to guess introduced by Poerner et al. (2020).”
OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, background): “While many works have been concentrating on injecting the homogeneous entity and relation knowledge from large scale knowledge graphs [36, 40, 51], none of them considers the problem of heterogeneous scenario which usually holds true in practice.”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “Particularly, certain relationship types let models exploit surface-level artifacts in subject entity names (Poerner et al., 2020; Cao et al., 2021).”
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, background): “Speciﬁcally, certain relationship types (e.g., nationalities) allow models to exploit surface-level artifacts in subject entity names (Poerner et al., 2020; Cao et al., 2021).”
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, unknown): “, 2019), E-BERT (Poerner et al., 2019)”