Neural Arithmetic Logic Units

By Andrew Trask, Felix Hill, Scott E. Reed, Jack W. Rae, Chris Dyer, Phil Blunsom. 2018

224 citations recorded by the source corpus.

Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training. To encourage more systematic numerical extrapolation, we propose an architecture that represents numerical quantities as linear activations which are manipulated using primitive arithmetic operators, controlled by learned gates. We call this module a neural arithmetic logic unit (NALU), by analogy to the arithmetic logic unit in traditional processors. Experiments show that NALU-enhanced neural networks can learn to track time, perform arithmetic over images of numbers, translate numerical language into real-valued scalars, execute computer code, and count objects in images. In contrast to conventional architectures, we obtain substantially better generalization both inside and outside of the range of numerical values encountered during training, often extrapolating orders of magnitude beyond trained numerical ranges.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Model Architectures

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…(Anonymous, 2019b), recursion (Cai et al., 2017), graph structure (Battaglia et al., 2018) or natural language (Andreas et al., 2018) in the latent space, differentiable external memory (Graves et al., 2016), or neural units designed to perform arbitrary arithmetic operations (Trask et al., 2018).”
Neural Execution Engines: Learning to Execute Subroutines (2020, background): “[25] explores the (poor) generalization capability of neural networks on scalar-values inputs outside of their training range, and develops new architectures that are better suited to scalar arithmetic, improving extrapolation.”
Weight Agnostic Neural Networks (2019, background): “Recent works offer a modern treatment [6] and application [61, 108] of these principles in the context of larger, deep neural network architectures.”
Deep Learning for Symbolic Mathematics (2019, methodology): “On these tasks, neural approaches tend to perform poorly, and require the introduction of components biased towards the task at hand (Kaiser & Sutskever, 2015; Trask et al., 2018).”
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019, methodology): “In developing the Neural Arithmetic Logic Unit, Trask et al. (2018) arrive at similar conclusions regarding extrapolation: neural models have difﬁculty outputting numerical values outside the training range.”
Quantity doesn’t buy quality syntax with neural language models (2019, neutral): “…known to struggle with generalizing short patterns to longer sequences, this pattern constitutes a failure to generalize to shorter sequences (cf. Trask et al., 2018); techniques for improving longer distance dependency learning in LMs (e.g., Trinh et al., 2018; Dai et al., 2019) are unlikely to…”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, methodology): “Special architectures have also been proposed for arithmetic tasks (Kaiser & Sutskever, 2015; Kalchbrenner et al., 2015; Price et al., 2016; Trask et al., 2018).”
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations (2019, background): “Recent progress in this vein of learning computational primitives includes addition/multiplication gates [41] and the Strassen 2× 2 matrix multiplication algorithm [42].”
Where’s My Head? Definition, Data Set, and Models for Numeric Fused-Head Identification and Resolution (2019, background): “Numeric expressions play an important role in various tasks, including textual entailment (Lev et al., 2004; Dagan et al., 2013), solving arithmetic problems (Roy and Roth, 2015), numeric reasoning (Roy et al., 2015; Trask et al., 2018) and language modeling (Spithourakis and Riedel, 2018).”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “Special architectures have also been proposed for arithmetic tasks (Kaiser & Sutskever, 2015; Kalchbrenner et al., 2015; Price et al., 2016; Trask et al., 2018).”
IsarStep: a Benchmark for High-level Mathematical Reasoning (2021, background): “However, research investigating their capacity of doing mathematical reasoning is still limited, with earlier attempts focusing on simple arithmetic tasks like integer addition and multiplication (Zaremba & Sutskever, 2014; Kaiser & Sutskever, 2016; Trask et al., 2018).”
Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks (2020, background): “The orthogonal direction of generalization to new numerical values is studied in [22, 26].”
Representing Numbers in NLP: a Survey and a Vision (2021, methodology): “, 2019) or modify activation functions to enhance numeracy (Trask et al., 2018).”
A Comprehensive Evaluation of Tool-Assisted Generation Strategies (2023, unknown): “There is adjacent research on augmenting neural networks, in ways besides textual interfaces, with tools (e.g., Andor et al., 2019; Jacovi et al., 2019) or training differentiable subnetworks that heavily mimic tools (Neelakantan et al., 2017; Trask et al., 2018).”
How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks (2020, background): “Another example is to encode log-and-exp transforms in the architecture to help extrapolate multiplication in arithmetic tasks (Trask et al., 2018; Madsen & Johansen, 2020).”
Learning Execution through Neural Code Fusion (2019, background): “Recent advances in representing and manipulating numbers for neural arithmetic logic units use scalar representations [40].”
Combinatorial optimization and reasoning with graph neural networks (2021, background): “Prominent examples include the neural GPU [63], neural RAM [72], neural programmer-intepreters [103] and neural arithmetic-logic units [128, 84].”
Neural Production Systems (2021, background): “Neural networks have been studied as a way to address the problems of learning procedural behavior and program induction (Graves et al., 2014; Reed & De Freitas, 2015; Neelakantan et al., 2015; Cai et al., 2017; Xu et al., 2018; Trask et al., 2018; Bunel et al., 2018; Li et al., 2020).”