Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation

By Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji. 2016

337 citations recorded by the source corpus.

Named Entity Disambiguation (NED) refers to the task of resolving multiple named entity mentions in a document to their correct references in a knowledge base (KB) (e.g., Wikipedia). In this paper, we propose a novel embedding method specifically designed for NED. The proposed method jointly maps words and entities into the same continuous vector space. We extend the skip-gram model by using two models. The KB graph model learns the relatedness of entities using the link structure of the KB, whereas the anchor context model aims to align vectors such that similar words and entities occur close to one another in the vector space by leveraging KB anchors and their context words. By combining contexts based on the proposed embedding with standard NED features, we achieved state-of-the-art accuracy of 93.1% on the standard CoNLL dataset and 85.2% on the TAC 2010 dataset.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Learning Dense Representations for Entity Retrieval (2019, neutral): “Yamada et al. (2016) use an alias table derived from the December 2014 Wikipedia dump, restricted to the ﬁfty most popular entities per mention.”
Autoregressive Entity Retrieval (2020, methodology): “Following previous works (Yamada et al., 2016; Ganea & Hofmann, 2017; Le & Titov, 2018), we considered only mentions that have entities in the KB (i.e., Wikipedia).”
ERNIE: Enhanced Language Representation with Informative Entities (2019, background): “In fact, some work has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results (Wang et al., 2014; Toutanova et al., 2015; Han et al., 2016; Yamada et al., 2016; Cao et al., 2017, 2018).”
Deep Joint Entity Disambiguation with Local Neural Attention (2017, methodology): “…(Huang et al., 2015) 86.6 (Ganea et al., 2016) 87.6 (Chisholm and Hachey, 2015) 88.7 (Guo and Barbosa, 2016) 89.0 (Globerson et al., 2016) 91.0 (Yamada et al., 2016) 91.5 our (global) 92.22 ± 0.14 ﬁrst train each entity vector on the entity canonical description page (title words included) for…”
Scalable Zero-shot Entity Linking with Dense Entity Retrieval (2019, methodology): “Prior work often used a ﬁxed number of k candidates where k ranges from 5 to 100 (for instance, Yamada et al. (2016) and Ganea and Hofmann (2017) choose k = 30 , (Lo-geswaran et al., 2019) choose k = 64 ).”
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2019, background): “Wikipedia2Vec (Yamada et al., 2016) embeds words and entities (Wikipedia URLs) into a common space.”
Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (2019, methodology): “Yamada et al. (2016, 2017) was the ﬁrst to investigate neural text representations and entity linking, but their approach is limited to ED.”
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA (2020, background): “(Poerner et al., 2019) injects factual entity knowledge into BERT’s embeddings without further training but by aligning Wikipedia2Vec entity vectors (Yamada et al., 2016) with BERT’s word piece vocabulary.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “E-BERT is developed by extending BERT model vocabulary with the Wikipedia2Vec entity vectors [187] after aligning.”
Boosting Entity Linking Performance by Leveraging Unlabeled Documents (2019, methodology): “We also compare to recent state-of-the-art systems trained supervisedly on Wikipedia and extra supervision or on AIDA CoNLL: Chisholm and Hachey (2015), Guo and Barbosa (2016), Glober-son et al. (2016), Yamada et al. (2016), Ganea and Hofmann (2017), Le and Titov (2018).”
Learning Dynamic Context Augmentation for Global Entity Linking (2019, background): “Many studies (Ratinov et al., 2011; Globerson et al., 2016; Yamada et al., 2016; Ganea and Hofmann, 2017; Le and Titov, 2018) obtain Φg by calculating all”
Learning Distributed Representations of Texts and Entities from Knowledge Base (2017, methodology): “They are shown to be effective for various KB-related tasks such as entity search (Hu et al., 2015), entity linking (Hu et al., 2015; Yamada et al., 2016), and link prediction (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015).”
CoLAKE: Contextualized Language and Knowledge Embedding (2020, methodology): “…of Mikolov et al. (2013) and Bordes et al. (2013), Wang et al. (2014) jointly learn embeddings for language and KG. Targeting mention-entity matching in entity linking, Yamada et al. (2016), Ganea and Hofmann (2017) also proposed joint methods to map entities and words into the same vector space.”
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020, methodology): “They include knowledge embeddings trained on knowledge graphs (Bor-des et al., 2013; Yang et al., 2015; Trouillon et al., 2016), and embeddings trained using textual contexts or descriptions of entities retrieved from a KB (Yamada et al., 2016, 2017; Cao et al., 2017; Ganea and Hofmann, 2017).”
Collective Entity Resolution with Multi-Focal Attention (2016, methodology): “With KB+HP*, we outperform Pershina et al. (2015), and are competitive with recent unpublished work by Yamada et al. (2016), which uses entity and word embeddings.”