Question-Answering with Grammatically-Interpretable Representations

By Hamid Palangi, Paul Smolensky, Xiaodong He, Li Deng. 2017

54 citations recorded by the source corpus.

We introduce an architecture, the Tensor Product RecurrentNetwork (TPRN). In our application of TPRN, internal representations—learned by end-to-end optimization in a deep neural network performing a textual question-answering(QA) task—can be interpreted using basic concepts from linguistic theory. No performance penalty need be paid for this increased interpretability: the proposed model performs comparably to a state-of-the-art system on the SQuAD QA task.The internal representation which is interpreted is a Tensor Product Representation: for each input word, the model selects a symbol to encode the word, and a role in which to place the symbol, and binds the two together. The selection is via soft attention. The overall interpretation is built from interpretations of the symbols, as recruited by the trained model, and interpretations of the roles as used by the model. We find support for our initial hypothesis that symbols can be interpreted as lexical-semantic word meanings, while roles can be interpreted as approximations of grammatical roles (or categories)such as subject, wh-word, determiner, etc. Fine-grained analysis reveals specific correspondences between the learned roles and parts of speech as assigned by a standard tagger(Toutanova et al. 2003), and finds several discrepancies in the model’s favor. In this sense, the model learns significant aspectsof grammar, after having been exposed solely to linguistically unannotated text, questions, and answers: no prior linguistic knowledge is given to the model. What is given is the means to build representations using symbols and roles, with an inductive bias favoring use of these in an approximately discrete manner.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, background): “…to enable the fully distributed representation of symbolic structure (like a hierarchy of features) which has recently resurfaced as an important direction for neural network research (Lake & Baroni, 2017; Bahdanau et al., 2018; van Steenkiste et al., 2019; Palangi et al., 2017; Tang et al., 2018).”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, background): “Here, the temporal logic rules are modelled in the form of recursive conjunctions represented by recurrent structures of RBMs. Temporal relational knowledge embedding has been studied recently in Tensor Product Recurrent Neural Network (TPRN) with applications to question-answering [32].”
Discovering the Compositional Structure of Vector Representations with Role Learning Networks (2019, methodology): “We add regularization similar to that in [15] to encourage one-hot vecotrs (Appendix A.3).”
Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering (2019, background): “A lot of research achievements are made centering on computer vision (CV) and natural language processing (NLP), especially for the task of visual question-answering (VQA) (Gao et al. 2018b; 2018a; Yang et al. 2016; Yu et al. 2017; Anderson et al. 2017; Palangi et al. 2018; Song et al. 2018).”
Machine Learning Interpretability: A Survey on Methods and Metrics (2019, background): “It is also worth mentioning existing works that make use of tensor product representation to perform knowledge encoding and logical reasoning based on common-sense inference [123], which is useful for increasing interpretability through question-answering with the model [124].”
Evidence Sentence Extraction for Machine Reading Comprehension (2019, methodology): “To improve the interpretability of question answering, previous work utilize interpretable internal representations (Palangi et al., 2017) or reasoning networks that employ a hop-by-hop reasoning process dynamically (Zhou et al., 2018).”
Learning Inference Rules with Neural TP-Reasoner (2020, methodology): “TPR ‘binding’ and ‘unbinding’ operations have been used in natural-language tasks(Palangi et al., 2018; Huang et al., 2018, 2019; Kezhen et al., 2020).”
emrQA: A Large Corpus for Question Answering on Electronic Medical Records (2018, background): “Roberts and Demner-Fushman (2016) Recent advances in QA combine logic-based and neural MC approaches to build hybrid models (Usbeck et al., 2015; Feng et al., 2016; Palangi et al., 2018).”
Mapping natural-language problems to formal-language solutions using structured neural representations (2019, unknown): “TPR binding has been used for encoding and exploring grammatical structural information of natural language (Palangi et al., 2018; Huang et al., 2019).”
RNNs Implicitly Implement Tensor Product Representations (2018, unknown): “While our work is orthogonal to this line of work, we note that TPRs and other ﬁller-role representations can proﬁtably be used as an explicit component of neural models (Koniusz et al., 2017; Palangi et al., 2018; Huang et al., 2018; Tang et al., 2018; Schlag & Schmidhuber, 2018).”
HUBERT Untangles BERT to Improve Transfer across NLP Tasks (2019, background): “In Palangi et al. (2018), a new structured recurrent unit (TPRN) was proposed to learn grammatically-interpretable representations using weak supervision from (context, question, answer) triplets in the SQuAD dataset (Rajpurkar et al., 2016).”
Natural- to formal-language generation using Tensor Product Representations (2019, methodology): “First, natural-language TPRs can be interpreted by exploring the distribution of tokens grouped by the role and symbol vectors they are assigned by a trained model (as in Palangi et al. (2018)).”