Leveraging Knowledge Bases in LSTMs for Improving Machine Reading

By Bishan Yang, Tom Michael Mitchell. 2017

258 citations recorded by the source corpus.

This paper focuses on how to take advantage of external knowledge bases (KBs) to improve recurrent neural networks for machine reading. Traditional methods that exploit knowledge from KBs encode knowledge as discrete indicator features. Not only do these features generalize poorly, but they require task-specific feature engineering to achieve good performance. We propose KBLSTM, a novel neural model that leverages continuous representations of KBs to enhance the learning of recurrent neural networks for machine reading. To effectively integrate background knowledge with information from the currently processed text, our model employs an attention mechanism with a sentinel to adaptively decide whether to attend to background knowledge and which information from KBs is useful. Experimental results show that our model achieves accuracies that surpass the previous state-of-the-art results for both entity extraction and event extraction on the widely used ACE2005 dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Knowledge Enhanced Contextual Word Representations (2019, background): “Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
DKN: Deep Knowledge-Aware Network for News Recommendation (2018, methodology): “These knowledge graphs are successfully applied in scenarios of machine reading[51], text classification[46], and word embedding[49].”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “These embeddings can not only help with the KG completion but also benefit various NLP applications (Yang and Mitchell, 2017; Zaremoodi et al., 2018; Han et al., 2018a).”
Differentiable Reasoning over a Virtual Knowledge Base (2020, background): “Knowledge Graph embeddings (Bordes et al., 2013; Yang et al., 2014; Dettmers et al., 2018) attach continuous representations to discrete symbols which allow them to be incorporated in deep networks (Yang & Mitchell, 2017).”
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (2018, neutral): “(Søgaard et al., 2014; Søgaard, 2013; Yang and Mitchell, 2017)) due to its improved power compared to the sign test variants.”
Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search (2020, methodology): “ knowledge base for response ranking in information seeking conversations. They proposed a model based on convolutional neural networks on top the interaction matrix. More recently, Yang and Mitchell [55] exploited knowledge bases to improve LSTM based networks for machine reading comprehension task”
Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision (2021, unknown): “Recent unsupervised MRC studies mainly adopt data synthetization approachs (Yang and Mitchell, 2017; Dhingra et al., 2018; Pan et al., 2020; Hong et al., 2020; Kang et al., 2020).”
Improving Question Answering over Incomplete KBs with Knowledge-Aware Reader (2019, background): “However, instead of applying a standard gating mechanism (Yang and Mitchell, 2017; Mihaylov and Frank, 2018), we propose a new conditional gating function that explicitly conditions on the question ~ q′.”
Exploiting the Ground-Truth: An Adversarial Imitation Based Knowledge Distillation Approach for Event Detection (2019, background): “To the end, the F 1 scores on English entities and event arguments are 83.4% and 65.7% respectively, which matches the state-of-the-arts (Yang and Mitchell 2017; Judea and Strube 2017).”
Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM (2018, methodology): “Most recently, neural sequential models (Ahn et al. 2016; Yang and Mitchell 2017) leverage the lower-dimensional continuous representation of knowledge concepts as additional inputs.”
Knowledge Distillation for Bilingual Dictionary Induction (2017, unknown): “First, due to advances in methods for extracting general purpose knowledge (Mitchell et al., 2015; Nakashole et al., 2013; Wijaya et al., 2014), the use of semantic knowledge has been explored for several natural language tasks (Nakashole and Mitchell, 2015; Yang and Mitchell, 2017).”
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (2019, methodology): “In this work, we treat WordNet synsets and NELL concepts as knowledge to be retrieved from KBs, similar to (Yang and Mitchell, 2017).”
RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space (2018, background): “Knowledge graphs are potentially useful to a variety of applications such as question-answering (Hao et al., 2017), information retrieval (Xiong et al., 2017), recommender systems (Zhang et al., 2016), and natural language processing (Yang & Mitchell, 2017).”
Relational Memory-Augmented Language Models (2022, background): “, 2021), machine reading (Yang and Mitchell, 2019), and reasoning (Minervini et al.”