Learning Continuous Semantic Representations of Symbolic Expressions

By Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, Charles Sutton. 2016

110 citations recorded by the source corpus.

Combining abstract, symbolic reasoning with continuous neural reasoning is a grand challenge of representation learning. As a step in this direction, we propose a new architecture, called neural equivalence networks, for the problem of learning continuous semantic representations of algebraic and logical expressions. These networks are trained to represent semantic equivalence, even of expressions that are syntactically very different. The challenge is that semantic representations must be computed in a syntax-directed manner, because semantics is compositional, but at the same time, small changes in syntax can lead to very large changes in semantics, which can be difficult for continuous neural architectures. We perform an exhaustive evaluation on the task of checking equivalence on a highly diverse class of symbolic algebraic and boolean expression types, showing that our model significantly outperforms existing architectures.

Subfields: Artificial Intelligence, Information Systems, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Representation & Self-Supervised Learning

Citations in the corpus

Analysing Mathematical Reasoning Abilities of Neural Models (2019, methodology): “Kaiser & Sutskever (2015) use an adapted convolutional architecture to solve addition and multiplication with good generalization; Allamanis et al. (2016) and Evans et al. (2018) use tree networks to predict polynomial or logical equivalence or logical entailment; Selsam et al. (2018) uses message…”
Relational inductive biases, deep learning, and graph networks (2018, background): “, 2016, 2017), algebraic and logical expressions (Allamanis et al., 2017; Evans et al., 2018), and programs (Devlin et al.”
Grammar Variational Autoencoder (2017, background): “Recently, Allamanis et al. (2016) learn representations for symbolic expressions from their parse trees.”
Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations (2017, methodology): “While not directly related to our work, Murali et al. [2017] perform computer program synthesis using grouped user-supplied example programs, and Allamanis et al. [2017] learn continuous semantic representations of mathematical and logical expressions.”
Deep Learning for Symbolic Mathematics (2019, background): “Allamanis et al. (2017) propose a framework called neural equivalence networks to learn semantic representations of algebraic expressions.”
Deep reinforcement learning with relational inductive biases (2018, background): “, 2018), learned approaches for inducing compositional programs (Reed & De Freitas, 2015; Parisotto et al., 2017; Allamanis et al., 2017; Devlin et al., 2017) and reasoning about structured data (Neelakantan et al.”
Learning to Prove Theorems via Interacting with Proof Assistants (2019, background): “A key ingredient is using a deep network to embed a program into a vector, by treating the program as a sequence of tokens (Allamanis et al., 2016b) or using structured input such as ASTs (Allamanis et al., 2016a; Alon et al., 2018) or graphs (Allamanis et al., 2017).”
Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization (2019, background): “The data sets used in research on this topic tend to be either fully formal, focusing on logic instead of natural language (Evans et al., 2018; Allamanis et al., 2016), or fully natural, as is the case for manually annotated data sets of English sentence pairs such as SICK (Marelli et al.”
Can Neural Networks Understand Logical Entailment? (2018, methodology): “The fourth and ﬁfth encoding benchmarks are (tree) recursive neural networks (Tai et al., 2015; Le & Zuidema, 2015; Zhu et al., 2015; Allamanis et al., 2016), also known as TreeRNNs.”
Learning to Perform Local Rewriting for Combinatorial Optimization (2018, methodology): “Notice that the grammar includes a more comprehensive operator set than previous works on finding equivalent expressions, which consider only boolean expressions [2, 17] or a subset of algorithmic operations [2].”
HOUDINI: Lifelong Learning as Program Synthesis (2018, background): “The framework of differentiable languages has proven especially powerful for representing architectures that have input-dependent structure, such as deep networks over trees [35, 2] and graphs [23, 19].”
Memory Augmented Recursive Neural Networks (2019, methodology): “…data in many applications including nat-ural scene classiﬁcation (Socher et al., 2011), sentiment classiﬁcation, Semantic Relatedness and syntactic parsing (Tai et al., 2015; Socher et al., 2011), neural programming and logic (Allamanis et al., 2017; Zaremba et al., 2014; Evans et al., 2018).”