Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

By Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, J. Urgen Schmidhuber, Jianfeng Gao. 2019

59 citations recorded by the source corpus.

We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-problems. The essential component of the model is a novel attention mechanism, called TP-Attention, which explicitly encodes the relations between each Transformer cell and the other cells from which values have been retrieved by attention. TP-Attention goes beyond linear combination of retrieved values, strengthening representation-building and resolving ambiguities introduced by multiple layers of standard attention. The TP-Transformer's attention maps give better insights into how it is capable of solving the Mathematics Dataset's challenging problems. Pretrained models and code will be made available after publication.

Subfields: Artificial Intelligence, Computer Science Applications, Developmental and Educational Psychology, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Developmental and Educational Psychology

In Natural Language Processing

In Model Architectures

Citations in the corpus

DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, methodology): “(1)In this sense, our model shares some similarity to Tensor Product Representation [23, 24, 25] where a word is represented using a tensor product of its filler (content) vector and its role (position) vector.”
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “…including Smolensky and Schmidhuber have produced better results on a mathematics problem set by combining BERT with a tensor products (Smolensky et al., 2016), a formal system for representing symbolic variables and their bindings (Schlag et al., 2019), creating a new system called TP-Transformer.”
Scaling Laws for Autoregressive Generative Modeling (2020, methodology): “The original authors trained various models, including transformers, on the math problem dataset [SGHK19], and it has also been studied with more specialized architectures [SSF19].”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, background): “In contrast, more general-purpose architectures fail to extrapolate on numerical tasks (Joulin & Mikolov, 2015; Dehghani et al., 2018; Schlag et al., 2019).”
Logic Tensor Networks (2020, background): “Integrative neural-symbolic approaches are known for either seeking to bring neurons into a symbolic model [41] or to bring symbols into neurons [61].”
A Reinforcement Learning Environment for Mathematical Reasoning via Program Synthesis (2021, background): “The predominant algorithmic approach for learning to produce the answers conditioned on the question statements has been to train seq2seq models [1][2][3].”
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2021, unknown): “Thus, greater novelty might be achieved through models that build in compositional mechanisms, such as RNNGs (Dyer et al., 2016) and TP-Transformers (Schlag et al., 2019).”
LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning (2021, background): “Schlag et al. (2019) developed an architecture for encoding relational information using tensor product representation for mathematical reasoning.”