Towards Neural Network-based Reasoning

By Baolin Peng, Zhengdong Lu, Hang Li, Kam-Fai Wong. 2015

75 citations recorded by the source corpus.

We propose Neural Reasoner, a framework for neural network-based reasoning over natural language sentences. Given a question, Neural Reasoner can infer over multiple supporting facts and find an answer to the question in specific forms. Neural Reasoner has 1) a specific interaction-pooling mechanism, allowing it to examine multiple facts, and 2) a deep architecture, allowing it to model the complicated logical relations in reasoning tasks. Assuming no particular structure exists in the question and facts, Neural Reasoner is able to accommodate different types of reasoning and different forms of language expressions. Despite the model complexity, Neural Reasoner can still be trained effectively in an end-to-end manner. Our empirical studies show that Neural Reasoner can outperform existing neural reasoning systems with remarkable margins on two difficult artificial tasks (Positional Reasoning and Path Finding) proposed in [8]. For example, it improves the accuracy on Path Finding(10K) from 33.4% [6] to over 98%.

Subfields: Artificial Intelligence, Model Architectures, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Question Answering & Reading Comprehension

Citations in the corpus

Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, background): “…models is their rich structure that enables matching and interaction between question, context, answer candidates and combinations thereof (Peng et al., 2015; Weissenborn, 2016; Xiong et al., 2016; Liu and Perez, 2017), which often is iterated over several times (Sordoni et al., 2016;…”
Neural Programmer: Inducing Latent Programs with Gradient Descent (2015, methodology): “Neelakantan et al. (2015); Lin et al. (2015) learn simple Horn clauses in a large knowledge base using RNNs. Neural networks have also been used for Q&A on datasets that do not require complicated arithmetic and logic reasoning (Bordes et al., 2014; Iyyer et al., 2014; Sukhbaatar et al., 2015; Peng ”
Dynamic Memory Networks for Visual and Textual Question Answering (2016, methodology): “We compare our best performing approach, DMN+, to two state of the art question answering architectures: the end to end memory network (E2E) (Sukhbaatar et al., 2015) and the neural reasoner framework (NR) (Peng et al., 2015).”
Adding Gradient Noise Improves Learning for Very Deep Networks (2015, background): “…performance, researchers have reported the necessity of additional techniques such as supervision in intermediate steps (Weston et al., 2014), warm-starts (Peng et al., 2015), random restarts, and the removal of certain activation functions in early stages of training (Sukhbaatar et al., 2015).”
End-to-end Differentiable Proving (2017, background): “Recent question-answering architectures such as [15, 17, 18] translate query representations implicitly in a vector space without explicit rule representations and can thus not easily incorporate domainspecific knowledge.”
NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language (2019, background): “…literature, such multi-hop reading comprehension tasks are frequently solved via end-to-end differentiable (deep learning) models (Sukhbaatar et al., 2015; Peng et al., 2015; Seo et al., 2016b; Raison et al., 2018; Henaff et al., 2016; Kumar et al., 2016; Graves et al., 2016; Dhingra et al., 2018).”
A Parallel-Hierarchical Model for Machine Comprehension on Sparse Data (2016, methodology): “The Neural Reasoner and the Attentive Reader are large, deep models with hundreds of thousands of parameters, so it is unsurprising that they performed poorly on MCTest.”
Learning Knowledge Base Inference with Neural Theorem Provers (2016, background): “Neural Reasoner (Peng et al., 2015) translates query representations in vector space without rule representations and can thus not incorporate domain-speciﬁc knowledge.”
End-To-End Memory Networks (2015, background): “(4)Following [17] we found adding more non-linearity solves tasks 17 and 19, see the supplementary material.”
A Deep Memory-based Architecture for Sequence-to-Sequence Learning (2015, background): “Although DEEPMEMORY is initially proposed for machine translation, it can be adapted for other tasks that require substantial transformations of sequences, including paraphrasing, reasoning (Peng et al., 2015), and semantic parsing (Yin et al.”
Attention-Based Convolutional Neural Network for Machine Comprehension (2016, methodology): “Peng et al. [ 2015 ] ’s Neural Reasoner infers over multiple supporting facts to generate an entity answer for a given question and it is also tested on bAbI.”
Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds (2018, background): “Various types of end-to-end neural networks (Sukhbaatar et al., 2015; Lee et al., 2015; Peng et al., 2015) have demonstrated promising accuracies on this dataset.”
Query-Reduction Networks for Question Answering (2016, methodology): “Compared to memory-based approaches (Weston et al., 2015; Sukhbaatar et al., 2015; Peng et al., 2015; Kumar et al., 2016; Xiong et al., 2016), QRN can better encodes locality information because it does not use a global memory access controller (circle nodes in Figure 2), and the query updates are…”
Visual question answering: A survey of methods and datasets (2016, background): “They were described in [42], with a number of variants proposed concurrently [83, 70, 8, 59].”
Learning a Natural Language Interface with Neural Programmer (2016, unknown): “Neural networks have also been applied on question-answering datasets that do not require much arithmetic reasoning (Bordes et al., 2014; Iyyer et al., 2014; Sukhbaatar et al., 2015; Peng et al., 2015; Hermann et al., 2015; Kumar et al., 2016).”
ListOps: A Diagnostic Dataset for Latent Tree Learning (2018, background): “The bAbI tasks have influenced the development of new learning algorithms (Sukhbaatar et al., 2015; Kumar et al., 2016; Peng et al., 2015).”