DeepType: Multilingual Entity Linking by Neural Type System Evolution

By Jonathan Raiman, Olivier Raiman. 2018

192 citations recorded by the source corpus.

The wealth of structured (e.g. Wikidata) and unstructured data about the world available today presents an incredible opportunity for tomorrow's Artificial Intelligence. So far, integration of these two different modalities is a difficult process, involving many decisions concerning how best to represent the information so that it will be captured or useful, and hand-labeling large amounts of data.DeepType overcomes this challenge by explicitly integrating symbolic information into the reasoning process of a neural network with a type system.First we construct a type system, and second, we use it to constrain the outputs of a neural network to respect the symbolic structure. We achieve this by reformulating the design problem into a mixed integer problem: create a type system and subsequently train a neural network with it. In this reformulation discrete variables select which parent-child relations from an ontology are types within the type system, while continuous variables control a classifier fit to the type system. The original problem cannot be solved exactly, so we propose a 2-step algorithm: 1) heuristic search or stochastic optimization over discrete variables that define a type system informed by an Oracle and a Learnability heuristic, 2) gradient descent to fit classifier parameters.We apply DeepType to the problem of Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC KBP 2010) and find that it outperforms all existing solutions by a wide margin, including approaches that rely on a human-designed type system or recent deep learning-based entity embeddings, while explicitly using symbolic information lets it integrate new entities without retraining.

Subfields: Artificial Intelligence, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Extraction

Citations in the corpus

Zero-shot Entity Linking with Dense Entity Retrieval (2019, background): “There is also recent work demonstrating that fine-grained entity typing information helps entity linking (Raiman and Raiman, 2018; Onoe and Durrett, 2019; Khalife and Vazirgiannis, 2018).”
Scalable Zero-shot Entity Linking with Dense Entity Retrieval (2019, background): “There is also recent work demonstrating that fine-grained entity typing information helps linking (Raiman and Raiman, 2018; Onoe and Durrett, 2019; Khalife and Vazirgiannis, 2018).”
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (2019, neutral): “However, most entity linking systems (Ganea and Hofmann, 2017; Raiman and Raiman, 2018) have been trained on”
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers (2020, background): “While reliable MT hinges on availability of large parallel corpora, transfer via multilingual KBs (Camacho-Collados et al., 2016; Mrkˇsi´c et al., 2017) is impaired by the limited KB coverage and inaccurate entity linking (Moro et al., 2014; Raiman and Raiman, 2018).”
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, methodology): “…for dealing with large vocabulary sizes (Morin and Bengio, 2005) in general, and large inventories of entities and entity types in particular (Raiman and Raiman, 2018; L´opez et al., 2019), we created a hierarchy of all entities in Wikidata, using a given entity’s position in this hierarchy…”
Evaluating the Impact of Knowledge Graph Context on Entity Disambiguation Models (2020, unknown): “do/Impact-of-KG-Context-on-ED Model In-KB. Acc. Yamada et al. (2016) [8] 91.5 Ganea&Hofmann (2017) [18] 92.22±0.14 Yang et al. (2018) [14] 93.0 Le&Titov (2018) [20] 93.07±0.27 DeepType (2018) [22] 94.88 Fang et al. (2019) [15] 94.3 Le& Titov (2019) [20] 89.66±0.16 DCA-SL (2019)[12] 94.64”
Learning Dynamic Context Augmentation for Global Entity Linking (2019, background): “Local EL methods disambiguate each mention independently according to their local contexts (Yamada et al., 2016; Chen et al., 2017; Globerson et al., 2016; Raiman and Raiman, 2018).”
The language of proteins: NLP, machine learning & protein sequences (2021, background): “between entities in knowledge graphs, and indeed improves performance [78,115].”
Learning Cross-Context Entity Representations from Text (2020, methodology): “On the TAC-KBP 2010 benchmark RELIC achieves 89.8% accuracy, just behind the top ranked system (Raiman & Raiman, 2018), which makes use of external knowledge bases, alias tables, and task-speciﬁc hand-engineered features.”
Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (2020, background): “As for entity linking, there are many mature entity linking systems for or factoid KGs, such as S-MART (Yang and Chang, 2015), DBpeida Lookup, and DeepType (Raiman and Raiman, 2018).”
Fine-Grained Entity Typing for Domain Independent Entity Linking (2019, background): “This represents a much larger-scale tagset than past work using entity typing for entity linking, which has usually used hundreds of types (Gupta et al., 2017; Murty et al., 2018; Raiman and Raiman, 2018).”