Deep Learning for Symbolic Mathematics

By Guillaume Lample, François Charton. 2019

500 citations recorded by the source corpus.

Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing these mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica.

Subfields: Artificial Intelligence, Hardware and Architecture, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Large Language Models

Citations in the corpus

Training Verifiers to Solve Math Word Problems (2021, background): “Other recent reasoning-related datasets have focused on mathematical reasoning on symbolic math (Lample and Charton, 2019), reading comprehension (LogiQA) (Liu et al., 2020), and commonsense question answering (Common-senseQA) (Talmor et al., 2018).”
Measuring Mathematical Problem Solving With the MATH Dataset (2021, methodology): “Lample and Charton (2020) use Transformers to solve algorithmically generated symbolic integration problems and achieve greater than 95% accuracy.”
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, methodology): “Conversely, some models that are pitched as neural networks, such as Lample and Charton's recent work on symbolic integration (Lample & Charton, 2019) turns out on careful inspection to have serious limits and to be heavily dependent on symbolic processors (Davis, 2019).”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, background): “, 2020), we do not design a new, ad-hoc module for neural reasoning, but we rely solely on the transformer’s capability to emulate algorithms (Wang et al., 2019b; Lample and Charton, 2020).”
Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “Regarding the two examples of mathematical problem solving mentioned in the previous paragraph, for example, [9] employed training set sizes of 20-40 million examples, and [3] used training sets with 2 million examples per module.”
Injecting Numerical Reasoning Skills into Language Models (2020, background): “A related line of work has been analyzing the mathematical reasoning abilities of neural models over text (Wallace et al., 2019; Rozen et al., 2019; Ravichander et al., 2019), and on arithmetic problems (Saxton et al., 2019; Amini et al., 2019; Lample and Charton, 2020).”
Transformers as Soft Reasoners over Language (2020, background): “…that transformers either already know [Talmor et al. , 2019] or can learn to emulate other algorithms, including for semantic parsing [He and Choi, 2019], machine translation [Wang et al. , 2019], symbolic integration [Lample and Charton, 2019], and mathematics [Saxton et al. , 2019].”
Generative Language Modeling for Automated Theorem Proving (2020, methodology): “Recently, Lample and Charton [39] successfully applied Transformers to anti-derivative calculus and solving differential equations, hinting that Transformers are capable of generating the exogenous terms involved in the substitutions required for successful symbolic integration.”
On the Binding Problem in Artificial Neural Networks (2020, background): “…et al., 2007; Hinton et al., 2012), and even in some inherently symbolic domains such as language modeling (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020), translation (Wu et al., 2016), board games (Silver et al., 2017), and symbolic integration (Lample and Charton, 2020).”
AudioLM: A Language Modeling Approach to Audio Generation (2022, background): “Generating natural signals with language models: Neural language models have demonstrated remarkable abilities for tasks as diverse as open-ended dialog modeling [35], code completion [36] or even solving integrals and differential equations [37].”
PRover: Proof Generation for Interpretable Reasoning over Rules (2020, background): “, 2019), symbolic integration (Lample and Charton, 2020) and mathematics (Saxton et al.”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, methodology): “Deep learning models were also successfully applied to symbolic integration, to solve differential equations (Lample & Charton, 2019), and automated theorem proving (Polu & Sutskever, 2020).”
GMAT: Global Memory Augmentation for Transformers (2020, methodology): “Moreover, Transformers with tailored attention patterns have been successfully used to replace convolutions in computer vision [16, 15], and have also been useful in music generation [7], symbolic mathematics [11] and other modalities.”
Symbolic Behaviour in Artificial Intelligence (2021, background): “Lample and Charton [5] even show that transformers in some cases outperform classical symbol systems like Mathematica that were specifically designed for these tasks, though there are certainly caveats to that particular comparison [82].”
Modelling High-Level Mathematical Reasoning in Mechanised Declarative Proofs (2020, background): “Sequence generation tasks have been proposed for mathematical reasoning [30, 44].”
NaturalProofs: Mathematical Theorem Proving in Natural Language (2021, background): “Lample and Charton [22] evaluate neural sequence models on symbolic integration problems, while Hendrycks et al. [16] propose a benchmark based on math competition problems.”