Learning Cross-Context Entity Representations from Text

By Jeffrey Ling, Nicholas FitzGerald, Zifei Shan, Livio Baldini Soares, Thibault Févry, David Weiss, Tom Kwiatkowski. 2020

33 citations recorded by the source corpus.

Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typing benchmarks, including a 64% error reduction over previous work on TypeNet (Murty et al., 2018); (2) a novel few-shot category reconstruction task; (3) existing entity linking benchmarks, where we match the state-of-the-art on CoNLL-Aida without linking-specific features and obtain a score of 89.8% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain training data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions such as: Who was the last inmate of Spandau jail in Berlin?

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “The concurrent “RELIC” and “EAE” models of Ling et al. (2020) and F´evry et al. (2020) learn representations for an explicitly predeﬁned set of entities and are evaluated on the same closed-book variant of TriviaQA that we consider.”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, methodology): “We compare our system to two paradigms for QA: Open-Book systems (ORQA and Graph Retriever), which ﬁrst retrieve a set of documents from a large corpus with the question and predict a span in the retrieved document as an answer and Closed-Book system ( RELIC and T5): model answers the question from ”
Explaining Question Answering Models through Text Generation (2020, background): “This has been demonstrated by their ability to answer questions that require common sense and world knowledge, without retrieving information from an external source (Trinh and Le, 2018; Zhou et al., 2019; Roberts et al., 2020; Ling et al., 2020).”
A Unified Encoder-Decoder Framework with Entity Memory (2022, methodology): “, 2020), and memory-based auto-encoder models like RELIC (Ling et al., 2020), EaE, and FILM (Verga et al.”
Reasoning Over Virtual Knowledge Bases With Open Predicate Relations (2021, methodology): “Entity Embeddings A global entity embedding table E ∈ R|E|×de is pretrained using the RELIC strategy (Ling et al., 2020); here E is the entity vocabulary and de is the embedding dimensionality.”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “The simplest representatives of this category train LMs to match individual to-kens (Broscheit, 2019) or mentions (Ling et al., 2020) in a text corpus to an entity bank, without any external resources.”
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020, methodology): “Similar to our pretraining task, NTEE (Yamada et al., 2017) and RELIC (Ling et al., 2020) use an approach that trains entity embeddings by predicting entities given their textual contexts obtained from a KB.”