Reasoning in Vector Space: An Exploratory Study of Question Answering

By Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, Paul Smolensky. 2015

39 citations recorded by the source corpus.

Question answering tasks have shown remarkable progress with distributed vector representation. In this paper, we investigate the recently proposed Facebook bAbI tasks which consist of twenty different categories of questions that require complex reasoning. Because the previous work on bAbI are all end-to-end models, errors could come from either an imperfect understanding of semantics or in certain steps of the reasoning. For clearer analysis, we propose two vector space models inspired by Tensor Product Representation (TPR) to perform knowledge encoding and logical reasoning based on common-sense inference. They together achieve near-perfect accuracy on all categories including positional reasoning and path finding that have proved difficult for most of the previous approaches. We hypothesize that the difficulties in these categories are due to the multi-relations in contrast to uni-relational characteristic of other categories. Our exploration sheds light on designing more sophisticated dataset and moving one step toward integrating transparent and interpretable formalism of TPR into existing learning paradigms.

Subfields: Artificial Intelligence, Natural Language Processing, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Question Answering & Reading Comprehension

Citations in the corpus

How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (2018, methodology): “Several other papers have discussed the weaknesses of various RC benchamrks (Chen et al., 2016; Lee et al., 2016).”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “…the authors put under scrutiny existing methodologies, datasets, or models were conducted by Callison-Burch et al. (2006, 2007); Tan et al. (2015); Post (2018) in machine translation, (Gkatzia and Mahamood, 2015; Reiter and Belz, 2009; Reiter, 2018) in natu-ral language generation, Lee et al. (2016”
Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds (2018, background): “Various types of end-to-end neural networks (Sukhbaatar et al., 2015; Lee et al., 2015; Peng et al., 2015) have demonstrated promising accuracies on this dataset.”
From Particular to General: A Preliminary Case Study of Transfer Learning in Reading Comprehension (2016, methodology): “The AS Reader (2)There is also a handcrafted model based on prior human analysis of the tasks [12] that solves all the task almost perfectly without any learning on training data.”
HUBERT Untangles BERT to Improve Transfer across NLP Tasks (2019, methodology): “In Lee et al. (2016), TPR-like representations were used to solve the bAbI tasks (Weston et al., 2016), achieving close to 100% accuracy in all but one of these tasks.”
Conditional Generative Adversarial Networks for Commonsense Machine Comprehension (2017, methodology): “However, this dataset is synthesized and only contains a small vocabulary, so a rule-based system solves them nearly totally correct [Lee et al., 2015].”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, background): “Near-neighbor Search is a well-studied problem with wide applications in recommendation system (Xue et al., 2017; Hall & Attenberg, 2015), question answering (Boytsov et al., 2016; Seo et al., 2019; Chang et al., 2020) and natural language processing (Bengio et al., 2003; Lee et al., 2016).”
Natural- to formal-language generation using Tensor Product Representations (2019, methodology): “Some researchers use TPRs for modeling deductive reasoning processes both on a rule-based model and deep learning models in vector space (Lee et al., 2016; Smolensky et al., 2016; Schlag & Schmidhuber, 2018).”
Learning Knowledge Base Inference with Neural Theorem Provers (2016, background): “…2014, NTMs), Memory Networks (Weston et al., 2015b), Neural Stacks/Queues (Grefenstette et al., 2015; Joulin and Mikolov, 2015), Neural Programmer (Neelakantan et al., 2016), Neural Programmer-Interpreters (Reed and de Freitas, 2016) and Hierarchical Attentive Memory (Andrychowicz and Kurach,…”
Basic Reasoning with Tensor Product Representations (2016, methodology): “In Lee et al. (2016), questions in category 5 are treated with an operation * that functions identically to ∘, with d × 2d matrix V rather than U.”
Mapping natural-language problems to formal-language solutions using structured neural representations (2019, methodology): “Some researchers use TPRs for modeling deductive reasoning processes both on a rule-based model and deep learning models in vector space (Lee et al., 2016; Smolensky et al., 2016; Schlag & Schmidhuber, 2018).”
Learning Inference Rules with Neural TP-Reasoner (2020, methodology): “Some researchers also use TPRs for modeling deductive reasoning processes both on a rule-based model and deep learning models in vector space (Lee et al., 2016; Smolensky et al., 2016; Schlag and Schmidhuber, 2018).”