Recursive Neural Networks Can Learn Logical Semantics

By Samuel R. Bowman, Christopher Potts, Christopher D. Manning. 2014

141 citations recorded by the source corpus.

Tree-structured recursive neural networks (TreeRNNs) for sentence meaning have been successful for many applications, but it remains an open question whether the fixed-length representations that they learn can support tasks as demanding as logical deduction. We pursue this question by evaluating whether two such models---plain TreeRNNs and tree-structured neural tensor networks (TreeRNTNs)---can correctly learn to identify logical relationships such as entailment and contradiction using these representations. In our first set of experiments, we generate artificial data from a logical grammar and use it to evaluate the models' ability to learn to handle basic relational reasoning, recursive structures, and quantification. We then evaluate the models on the more natural SICK challenge data. Both models perform competitively on the SICK data and generalize well in all three experiments on simulated data, suggesting that they can learn suitable representations for logical inference in natural language.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Learning to Execute (2014, methodology): “There has been related research that used Tree Neural Network (sometimes known as Recursive Neural Networks) to evaluate symbolic mathematical expressions and logical formulas (Zaremba et al., 2014a; Bowman et al., 2014; Bowman, 2013), which is close in spirit to our work.”
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (2017, positive): “The TreeRNN models are trained using a training regime inspired by Bowman et al. (2015b).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “[108] proposed to classify the logical relationship between sentences with recursive neural networks.”
Traversing Knowledge Graphs in Vector Space (2015, background): “Previous works have explored compositional vector space representations in the context of sentence interpretation (Socher et al., 2012; Socher et al., 2014; Grefenstette, 2013; Bowman et al., 2014).”
Lifted Rule Injection for Relation Embeddings (2016, background): “implication rules, is in the focus of current research (Rocktäschel et al., 2014; Wang et al., 2014; Bowman et al., 2015; Wang et al., 2015; Vendrov et al., 2016; Hu et al., 2016; Rocktäschel and Riedel, 2016; Cohen, 2016).”
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018, background): “Bowman et al. (2014; 2015) also demonstrate that these recursive structures work better for downstream, predictive tasks if the data was generated with such a structure.”
Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization (2019, methodology): “The data generation process is inspired by Bowman et al. (2015b): an artificial language is defined, sentences are generated according to its grammar and the entailment relation between pairs of such sentences is established according to a fixed background logic.”
Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences (2018, methodology): “This makes its learning task much simpler: it need only determine these local relations and propagate them; we know from Bowman et al. (2015b) that the relational algebra that underlies this propagation can easily be learned by standard neural architectures.”
When Are Tree Structures Necessary for Deep Learning of Representations? (2015, background): “…have been proposed (e.g., (Paulus et al., 2014; Irsoy and Cardie, 2014)), and applied to various NLP tasks, among them entailment (Bowman, 2013; Bowman et al., 2014), sentiment analysis (Socher et al., 2013; Irsoy and Cardie, 2013; Dong et al., 2014), question-answering (Iyyer et al., 2014),…”
A Benchmark for Systematic Generalization in Grounded Language Understanding (2020, background): “Recent work has recognized the advantages of compositional generalization for robustness and sample efficiency, and responded by building synthetic environments to evaluate models on aspects of this skill [7, 21, 25, 30, 4, 16, 17, 8, 19, 3].”
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (2020, background): “In particular, Bowman et al. (2015) showed that tree-structured neural networks generalized to longer sequences.”
Can Neural Networks Understand Monotonicity Reasoning? (2019, background): “The results show that all models trained with SNLI (Bowman et al., 2015b) and MultiNLI (Williams et al., 2018) perform worse on downward inferences than on up-ward inferences.”
A Natural Proof System for Natural Language (2017, methodology): “7 Recent developments in compositional distributional semantics saw several new approaches to RTE that employ artificial neural networks for machine learning (Bowman et al., 2015b).”