Learning Distributed Representations of Texts and Entities from Knowledge Base

By Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji. 2017

103 citations recorded by the source corpus.

We describe a neural network model that jointly learns distributed representations of texts and knowledge base (KB) entities. Given a text in the KB, we train our proposed model to predict entities that are relevant to the text. Our model is designed to be generic with the ability to address various NLP tasks with ease. We train the model using a large corpus of texts and their entity annotations extracted from Wikipedia. We evaluated the model on three important NLP tasks (i.e., sentence textual similarity, entity linking, and factoid question answering) involving both unsupervised and supervised settings. As a result, we achieved state-of-the-art results on all three of these tasks. Our code and trained models are publicly available for further academic research.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Improving Question Answering with External Knowledge (2019, methodology): “nchor links in Wikipedia. Similarity refers to the context similarity between mention-entity pairs. We adopt a neural network model that jointly learns embedding of words and entities from Wikipedia (Yamada et al., 2017). For each entity mention m, we build the vector representation of its context v”
Learning Dense Representations for Entity Retrieval (2019, neutral): “, 2016; Ganea and Hofmann, 2017), joint modeling with related tasks like textual similarity (Yamada et al., 2017b; Barrena et al., 2018) and cross-lingual modeling (Sil et al.”
Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (2019, methodology): “Yamada et al. (2016, 2017) was the ﬁrst to investigate neural text representations and entity linking, but their approach is limited to ED.”
Learning Cross-Context Entity Representations from Text (2020, methodology): “In Section 5.4, we show that Yamada et al. (2017)’s entity embeddings are good at capturing broad entity types such as Tennis Player but less good at capturing more complex compound types such as Scottish Footballers .”
Entity Commonsense Representation for Neural Abstractive Summarization (2018, methodology): “Third, we can make use of the entity common-sense that can be learned as a continuous vector representation from a separate larger corpus (Ni et al., 2016; Yamada et al., 2017).”
Joint Learning of Named Entity Recognition and Entity Linking (2019, methodology): “The entity embeddings are 300 dimensional and were trained by Yamada et al. (2017) on Wikipedia.”
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020, methodology): “Similar to our pretraining task, NTEE (Yamada et al., 2017) and RELIC (Ling et al., 2020) use an approach that trains entity embeddings by predicting entities given their textual contexts obtained from a KB.”
Fine-Grained Entity Typing for Domain Independent Entity Linking (2019, background): “Learning pretrained entity representations from knowledge bases has also been studied for entity linking (Hu et al., 2015; Yamada et al., 2016, 2017; Eshel et al., 2017).”
DeepType: Multilingual Entity Linking by Neural Type System Evolution (2018, methodology): “2016) and (Yamada et al. 2017), the authors adopt a distributed representation of context which either models words and entities, or documents and entities such that distances between vectors informs disambiguation.”
Entity-aware ELMo: Learning Contextual Entity Representation for Entity Disambiguation (2019, methodology): “Yamada et al. (2016; 2017) and Ganea et al. (2017) learn entity representations by encoding the co-occurrence statistics between the entity and the words in its context.”
Improving Entity Linking by Modeling Latent Relations between Mentions (2018, methodology): “As with other recent approaches to NEL (Yamada et al., 2017; Ganea and Hofmann, 2017), we rely on representation learning and learn embeddings of mentions, contexts and relations.”