Dynamic Integration of Background Knowledge in Neural NLU Systems

By Dirk Weissenborn, Tom'avs Kovcisk'y, Chris Dyer. 2017

62 citations recorded by the source corpus.

Common-sense or background knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, the requisite background knowledge is indirectly acquired from static corpora. We develop a new reading architecture for the dynamic integration of explicit background knowledge in NLU models. A new task-agnostic reading module provides refined word representations to a task-specific NLU architecture by processing background knowledge in the form of free-text statements, together with the task-specific inputs. Strong performance on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) demonstrate the effectiveness and flexibility of our approach. Analysis shows that our models learn to exploit knowledge selectively and in a semantically appropriate way.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “…47.06 49.54 55.80 MEMEN* (Pan et al., 2017) 43.16 46.90 49.28 55.83 Mnemonic Reader (Hu et al., 2017) 46.94 52.85 54.45 59.46 Reading Twice for NLU (Weissenborn et al., 2017a) approaches as the model is used to independently process an increasing number of these paragraphs and the model’s most…”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, methodology): “Previous work has converted KB facts to sentences to provide extra evidence (Weissenborn et al., 2017; Mihaylov and Frank, 2018), but do not explicitly use the KB graph structure.”
Unsupervised Question Answering by Cloze Translation (2019, background): “Finally, injecting external knowledge into QA systems could be viewed as semi-supervision, and Weissenborn et al. (2017) and Mihaylov and Frank (2018) use Conceptnet (Speer et al., 2016) for QA tasks.”
Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge (2018, background): “In contrast to the implicit knowledge transfer of Weissenborn et al. (2017), our explicit attention over external knowledge facts can deliver insights about the used knowledge and how it interacts with specific context tokens (see Section 6).”
NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language (2019, methodology): “Moreover, such models tend to require large amounts of training data to generalise correctly, and incorporating background knowledge is still an open problem (Rocktäschel et al., 2015; Weissenborn et al., 2017a; Rocktäschel and Riedel, 2017; Evans and Grefenstette, 2017).”
Explicit Utilization of General Knowledge in Machine Reading Comprehension (2018, methodology): “The existing way to solve the second problem is to encode general knowledge in vector space so that the encoding results can be used to enhance the lexical or contextual representations of words (Weissenborn et al., 2017; Mihaylov and Frank, 2018).”
Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering (2019, background): “Weissenborn et al. (2018) proposed a strategy of dynamically reﬁning word embeddings by reading input text as well as external knowledge, such as ConceptNet and Wikipedia abstracts.”
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (2019, background): “Weissenborn et al. (2017), Bauer et al. (2018), Mihaylov and Frank (2018), Pan et al. (2019), Chen et al. (2018), Wang et al. (2018) follow a retrieve-then-encode paradigm, i.e., they ﬁrst retrieve relevant knowledge from KBs, and only the retrieved knowledge relevant locally to the reading text…”
Reasoning Over Virtual Knowledge Bases With Open Predicate Relations (2021, methodology): “…that use KB triples have been with only a few hundreds of triples in the model at any point, necessitating a separate heuristic process to retrieve candidate KB triples (Ahn et al., 2016; Henaff et al., 2016; Weissenborn et al., 2017; Chen et al., 2018; Mihaylov & Frank, 2018; Logan et al., 2019a).”
Incorporating External Knowledge into Machine Reading for Generative Question Answering (2019, background): “2. gQA w/ CRWE (Weissenborn, 2017): CRWE is a reading architecture with dynamic integration of background knowledge based on contextual reﬁnement of word embeddings by leveraging supplementary knowledge.”
Machine Common Sense Concept Paper (2018, background): “6 Approved for Public Release, Distribution Unlimited These researchers are discovering new techniques for extracting commonsense knowledge from language [42][43][44][45], vision [46][47][48][49], and robotics [50][51][52].”
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference (2018, background): “ a copying mechanism over fact attributes. Yang and Mitchell (2017) modify the LSTM cell to incorporate WordNet and NELL knowledge for event and entity extraction. For cross-sentence inference tasks, Weissenborn et al. (2018) and Mihaylov and Frank (2018) dynamically reﬁne word representations by re”
A dataset and baselines for sequential open-domain question answering (2018, result): “Our baseline, which improves reading by incorporating additional relation description spans, is similar to Weissenborn et al. (2017) and Mihaylov and Frank (2018), who integrate background commonsense knowledge into reading-comprehension systems.”
Commonsense for Generative Multi-Hop Question Answering Tasks (2018, methodology): “In MRC, Weissenborn et al. (2017) integrated external background knowledge into an NLU model by using contextually-reﬁned word embeddings which integrated information from Con-ceptNet (single-hop relations mapped to unstructured text) via a single layer bidirectional LSTM. Concurrently to our work, ”