Linguistic Knowledge as Memory for Recurrent Neural Networks

By Bhuwan Dhingra, Zhilin Yang, William W. Cohen, Ruslan Salakhutdinov. 2017

37 citations recorded by the source corpus.

Training recurrent neural networks to model long term dependencies is difficult. Hence, we propose to use external linguistic knowledge as an explicit signal to inform the model which memories it should utilize. Specifically, external knowledge is used to augment a sequence with typed edges between arbitrarily distant elements, and the resulting graph is decomposed into directed acyclic subgraphs. We introduce a model that encodes such graphs as explicit memory in recurrent neural networks, and use it to model coreference relations in text. We apply our model to several text comprehension tasks and achieve new state-of-the-art results on all considered benchmarks, including CNN, bAbi, and LAMBADA. On the bAbi QA tasks, our model solves 15 out of the 20 tasks with only 1000 training examples per task. Analysis of the learned representations further demonstrates the ability of our model to encode fine-grained entity information across a document.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains (2019, background): “Kishimoto et al. (2018) fed external world knowledge (ConceptNet relations and coreferences) explicitly into MAGE-GRU (Dhingra et al., 2017) and achieved improvements compared to only us-ing the relational arguments.”
Reading Twice for Natural Language Understanding (2017, methodology): “[6] exploit linguistic knowledge using MAGE-GRUs, an adapation of GRUs to handle graphs, however, external knowledge has to be present in form of triples.”
On-The-Fly Information Retrieval Augmentation for Language Models (2020, background): “…demonstrated for NLP tasks such as natural language inference (Chen et al., 2018; Yang et al., 2019), language generation (Parthasarathi and Pineau, 2018), knowledge base completion (Toutanova et al., 2015; Das et al., 2017) and question answering (Sun et al., 2019, 2018; Dhingra et al., 2017).”
Neural Models for Reasoning over Multiple Mentions Using Coreference (2018, methodology): “Part of this work was described in an unpub-Mary lished preprint (Dhingra et al., 2017b).”
Dynamic Integration of Background Knowledge in Neural NLU Systems (2017, background): “…understanding has long been remarked on, especially in the context of classical models of AI (Schank & Abelson, 1977; Minsky, 2000); however, it has only recently begun to play a role in neural network models of NLU (Ahn et al., 2016; Xu et al., 2016; Long et al., 2017; Dhingra et al., 2017).”
An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks (2017, background): “In other work (Dhingra et al., 2017), information about long-term dependencies are also represented in the form of symbolic knowledge to improve the ability of RNNs to handle long-term text data.”
Robustness in Coreference Resolution (2020, background): “…recently, e.g. in machine translation (Hardmeier et al., 2015; Guillou et al., 2016), question answering (Choi et al., 2017), text compression (Dhingra et al., 2017), text summarization (Durrett et al., 2016), slot-filling (Yu & Ji, 2016), math problem solving (Matsuzaki et al., 2017), or…”
Evidence Sentence Extraction for Machine Reading Comprehension (2019, methodology): “Besides widely used lexical features such as part-of-speech tags and named entity types (Wang et al., 2016; Liu et al., 2017; Dhingra et al., 2017b, 2018), we consider more diverse types of external knowledge for performance improvements.”