Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

By Samuel Broscheit. 2019

138 citations recorded by the source corpus.

A typical architecture for end-to-end entity linking systems consists of three steps: mention detection, candidate generation and entity disambiguation. In this study we investigate the following questions: (a) Can all those steps be learned jointly with a model for contextualized text-representations, i.e. BERT? (b) How much entity knowledge is already contained in pretrained BERT? (c) Does additional entity knowledge improve BERT’s performance in downstream tasks? To this end we propose an extreme simplification of the entity linking setup that works surprisingly well: simply cast it as a per token classification over the entire entity vocabulary (over 700K classes in our case). We show on an entity linking benchmark that (i) this model improves the entity representations over plain BERT, (ii) that it outperforms entity linking architectures that optimize the tasks separately and (iii) that it only comes second to the current state-of-the-art that does mention detection and entity disambiguation jointly. Additionally, we investigate the usefulness of entity-aware token-representations in the text-understanding benchmark GLUE, as well as the question answering benchmarks SQUAD~V2 and SWAG and also the EN-DE WMT14 machine translation benchmark. To our surprise, we find that most of those benchmarks do not benefit from additional entity knowledge, except for a task with very small training data, the RTE task in GLUE, which improves by 2%.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “Additionally, recent works (Petroni et al., 2019; Logan et al., 2019; Broscheit, 2019; Roberts et al., 2020) demonstrate that beyond general linguistic capabilities, language models can also pick up factual knowledge present in the training data.”
Autoregressive Entity Retrieval (2020, methodology): “…work on entity retrieval (e.g. Hoffart et al., 2011; Pic-cinno & Ferragina, 2014; Huang et al., 2015; Le & Titov, 2018; Logeswaran et al., 2019; Broscheit, 2019; Wu et al., 2019, to name just a few) there is a common design choice to most current solutions: entities are associated with a…”
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2019, background): “While there are recent advances in fully end-to-end EL (Broscheit, 2019), the task is typically broken down into three steps: (1) detecting spans that are potential entity spans, (2) generating sets of candidate entities for these spans, (3) selecting the correct candidate for each span.”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “Without external information In the simplest case, Broscheit (2019) shows that LMs can be”