Can Neural Networks Understand Logical Entailment?

By Richard Evans, David Saxton, David Amos, Pushmeet Kohli, Edward Grefenstette. 2018

139 citations recorded by the source corpus.

We introduce a new dataset of logical entailments for the purpose of measuring models' ability to capture and exploit the structure of logical expressions against an entailment prediction task. We use this task to compare a series of architectures which are ubiquitous in the sequence-processing literature, in addition to a new model class---PossibleWorldNets---which computes entailment as a "convolution over possible worlds". Results show that convolutional networks present the wrong inductive bias for this class of problems relative to LSTM RNNs, tree-structured neural networks outperform LSTM RNNs due to their enhanced ability to exploit the syntax of logic, and PossibleWorldNets outperform all benchmarks.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Analysing Mathematical Reasoning Abilities of Neural Models (2019, methodology): “…(2015) use an adapted convolutional architecture to solve addition and multiplication with good generalization; Allamanis et al. (2016) and Evans et al. (2018) use tree networks to predict polynomial or logical equivalence or logical entailment; Selsam et al. (2018) uses message passing…”
Relational inductive biases, deep learning, and graph networks (2018, background): “We are optimistic about a number of other relevant, and perhaps underappreciated, research directions, including marrying learning-based approaches with programs (Ritchie et al., 2016; Andreas et al., 2016; Gaunt et al., 2016; Evans and Grefenstette, 2018; Evans et al., 2018), developing model-based”
The Importance of Being Recurrent for Modeling Hierarchical Structure (2018, methodology): “Concurrently to our work Evans et al. (2018) proposed an alternative data set for logical inference and also found that a FAN model underperformed various other architectures including LSTMs.”
Learning a SAT Solver from Single-Bit Supervision (2018, background): “Evans et al. (2018) present a neural network architecture that can learn to predict whether one propositional formula entails another by randomly sampling and evaluating candidate assignments.”
Improving Graph Neural Network Representations of Logical Formulae with Subgraph Pooling (2019, background): “Other conceptually-related works in this area include: 1) [20] where the authors introduced a dataset for evaluating neural models on logical entailment and explored the use of several popular neural architectures (e.g. Tree LSTMs, BiDirLSTMs, etc.) on the proposed task.”
Learning to Perform Local Rewriting for Combinatorial Optimization (2018, background): “Notice that the grammar includes a more comprehensive operator set than previous works on finding equivalent expressions, which consider only boolean expressions [2, 17] or a subset of algorithmic operations [2].”
A Simple Recurrent Unit with Reduced Tensor Product Representations (2018, methodology): “Symbolic Vocabulary Permutation (Evans et al., 2018) is applied as data augmentation during learning on the logical entailment task: exploiting α -equivalence, this systematically replaces the variables in a proposition pair with randomly sampled alternative variables, as only connectives matter on…”
Deep reinforcement learning with relational inductive biases (2018, background): “, 2016), or even explicit logical reasoning over structured internal representations (Evans et al., 2018), drawing inspiration from more symbolic approaches in classic AI.”
Can Neural Networks Understand Monotonicity Reasoning? (2019, background): “A number of recent works attempt to test and analyze what type of inferences an NLI model may be performing, focusing on various types of lexical inferences (Glockner et al., 2018; Naik et al., 2018; Poliak et al., 2018) and logical inferences (Bowman et al., 2015b; Evans et al., 2018).”
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (2020, result): “For structural generalization cases, the results of Bowman et al. (2015); Evans et al. (2018) and McCoy et al. (2019) suggest that tree-structured models may provide a better inductive bias.”
Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations (2018, methodology): “The logical entailment task was introduced recently, and a model [16] was proposed that used given 77 parse trees of the input propositions and was designed specifically for the task.”
FHI Oxford Technical Report # 2018-2 Predicting Human Deliberative Judgments with Machine Learning (2018, background): “Second, deliberation can be modeled explicitly by building on recent deep learning approaches [28, 29, 30, 31, 32, 33, 34, 35, 36, 37].”
Analyzing Compositionality-Sensitivity of NLI Models (2018, methodology): “structure of sentences. However, past studies have shown that RNNs (especially sequential RNNs) are insufﬁcient for effectively capturing logical compositional structure that is often present in NLI (Evans et al. 2018; Nangia and Bowman 2018). These indicate an over-focus on lexical information in n”
Proof Artifact Co-training for Theorem Proving with Language Models (2021, background): “More recently, progress has been made in directly applying deep learning to prove theorems (Rocktäschel & Riedel, 2017; Evans et al., 2018; Selsam et al., 2019; Lederman et al., 2020).”
IsarStep: a Benchmark for High-level Mathematical Reasoning (2021, background): “, 2019), logical reasoning (Evans et al., 2018), and problems of function integration, ordinary differential equations (Lample & Charton, 2020), and properties of differential systems (Charton et al.”
LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning (2021, methodology): “Initial works focused on encoding the tree structure of expressions using TreeRNNs (Evans et al., 2018).”
Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization (2019, background): “The data sets used in research on this topic tend to be either fully formal, focusing on logic instead of natural language (Evans et al., 2018; Allamanis et al., 2016), or fully natural, as is the case for manually annotated data sets of English sentence pairs such as SICK (Marelli et al.”
Memory Augmented Recursive Neural Networks (2019, methodology): “…data in many applications including nat-ural scene classiﬁcation (Socher et al., 2011), sentiment classiﬁcation, Semantic Relatedness and syntactic parsing (Tai et al., 2015; Socher et al., 2011), neural programming and logic (Allamanis et al., 2017; Zaremba et al., 2014; Evans et al., 2018).”