Learning to Reason with Third-Order Tensor Products

By Imanol Schlag, J. Urgen Schmidhuber. 2018

70 citations recorded by the source corpus.

We combine Recurrent Neural Networks with Tensor Product Representations to learn combinatorial representations of sequential data. This improves symbolic interpretation and systematic generalisation. Our architecture is trained end-to-end through gradient descent on a variety of simple natural language reasoning tasks, significantly outperforming the latest state-of-the-art models in single-task and all-tasks settings. We also augment a subset of the data such that training and test data exhibit large systematic differences and show that our approach generalises better than the previous state-of-the-art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Statistical Learning Theory

Citations in the corpus

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, background): “This embedding is a Tensor-Product Representation ( TPR ; Smolensky, 1990) in an end-to-end-differentiable TPR system (Schlag & Schmidhuber, 2018; Schmidhuber, 1992) that learns “internal spotlights of attention” (Schmidhu-ber, 1993).”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “In the recent literature, TPR-like mechanisms have been incorporated into neural networks using fast-weights (Schlag and Schmidhuber, 2018) or self-attention (Schlag et al.”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “Fast weights have also been employed in a variety of SL contexts [e.g. Ha et al., 2016; Jia et al., 2016; Bertinetto et al., 2016; Schlag and Schmidhuber, 2018].”
Linear Transformers Are Secretly Fast Weight Programmers (2021, background): “Ba et al. (2016) have already pointed out a relation between a variant of outer product-based FWPs (Schmidhuber, 1993) and attention (Bahdanau et al., 2015).”
RNNs Implicitly Implement Tensor Product Representations (2018, unknown): “While our work is orthogonal to this line of work, we note that TPRs and other ﬁller-role representations can proﬁtably be used as an explicit component of neural models (Koniusz et al., 2017; Palangi et al., 2018; Huang et al., 2018; Tang et al., 2018; Schlag & Schmidhuber, 2018).”
A Simple Recurrent Unit with Reduced Tensor Product Representations (2018, methodology): “…(2018), which performed question-answering by using TPRs to embed simple knowledge graphs as tensors; our model does not deploy graph-based inference, which is less natural for the inference tasks we address than it is for the question-answering task addressed in Schlag & Schmidhuber (2018).”
Neurosymbolic AI: the 3rd wave (2020, unknown): “Another very relevant application domain is planning, which requires learning and reasoning over time, as in this example adapted from Schlag and Schmidhuber (2018): Daniel picks up the milk; Daniel goes to the bedroom; Daniel places the milk on the table; Daniel goes to the bathroom.”
Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations (2018, background): “e binding and unbinding operations are explicitly carried out in our proposed recurrent unit, and the two operations directly determine the update of the hidden states. Closest to the present work is [31], which performed question-answering by using TPRs to embed simple knowledge graphs as tensors; ”
Natural- to formal-language generation using Tensor Product Representations (2019, methodology): “Some researchers use TPRs for modeling deductive reasoning processes both on a rule-based model and deep learning models in vector space (Lee et al., 2016; Smolensky et al., 2016; Schlag & Schmidhuber, 2018).”
Learning Inference Rules with Neural TP-Reasoner (2020, methodology): “Some researchers also use TPRs for modeling deductive reasoning processes both on a rule-based model and deep learning models in vector space (Lee et al., 2016; Smolensky et al., 2016; Schlag and Schmidhuber, 2018).”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, methodology): “Other approaches for solving reasoning tasks end-to-end include Goyal et al. (2021), Seraﬁni & d’Avila Garcez (2016), and Schlag & Schmidhuber (2018).”
Logic Tensor Networks (2020, background): “[60], which show that state-of-the-art recurrent neural networks may fail at simple question-answering tasks, despite achieving very high accuracy.”
A Perspective on Objects and Systematic Generalization in Model-Based RL (2019, background): “Schmidhuber (1992; 1993); Schlag & Schmidhuber (2018)) it may be more easy to learn general functions of this kind.”