When Are Tree Structures Necessary for Deep Learning of Representations?

By Jiwei Li, Thang Luong, Dan Jurafsky, Eduard H. Hovy. 2015

230 citations recorded by the source corpus.

Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture. However there have not been rigorous evaluations showing for exactly which tasks this syntax-based method is appropriate. In this paper, we benchmark recursive neural models against sequential recurrent neural models, enforcing applesto-apples comparison as much as possible. We investigate 4 tasks: (1) sentiment classification at the sentence level and phrase level; (2) matching questions to answerphrases; (3) discourse parsing; (4) semantic relation extraction. Our goal is to understand better when, and why, recursive models can outperform simpler models. We find that recursive models help mainly on tasks (like semantic relation extraction) that require longdistance connection modeling, particularly on very long sequences. We then introduce a method for allowing recurrent models to achieve similar performance: breaking long sentences into clause-like units at punctuation and processing them separately before combining. Our results thus help understand the limitations of both classes of models, and suggest directions for improving recurrent models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

A Fast Unified Model for Parsing and Sentence Understanding (2016, background): “TreeRNNs have shown promise (Tai et al., 2015; Li et al., 2015; Bowman et al., 2015b), but have largely been overlooked in favor of sequence-based RNNs because of their incompatibility with batched computation and their reliance on external parsers.”
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation (2015, methodology): “Our compositional character to word (C2W) model is based on bidirectional LSTMs (Graves and Schmidhuber, 2005), which are able to learn complex non-local dependencies in sequence models.”
End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures (2016, methodology): “We follow the settings by (Li and Ji, 2014), and we did not use the full mention boundary unlike Lu and Roth (2015).”
What Do Recurrent Neural Network Grammars Learn About Syntax? (2016, background): “Li et al. (2016), where sequential LSTMs discover polarity information in sentiment analysis, although perhaps more surprising as polarity information is less intuitively central to syntax and language modeling.”
Tree-Structured Composition in Neural Networks without Tree-Structured Architectures (2015, background): “Nevertheless, tree models have not shown the kinds of dramatic performance improvements over sequence models that their billing would lead one to expect: head-to-head comparisons with sequence models show either modest improvements (Tai et al., 2015) or none at all (Li et al., 2015).”
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding (2017, background): “The idea of using both forward and backward attentions is inspired by Bi-directional LSTM (Bi-LSTM) (Graves, Jaitly, and Mohamed 2013), in which forward and backward LSTM are used to encode long-range dependency.”
A C-LSTM Neural Network for Text Classification (2015, background): “In (Li et al., 2015), the authors suggest that sequence-based models are sufficient to capture the compositional semantics for many NLP tasks, thus in this work the CNN is directly built upon word sequences other than the syntactic parse tree.”
BP-Transformer: Modelling Long-Range Context via Binary Partitioning (2019, neutral): “We use pre-trained GloVe embedding (Pennington et al., 2014) (Li et al., 2015) 49.8 - Tree-LSTM (Socher et al., 2013) 51.0 - QRNN (Bradbury et al., 2017) - 91.4 BCN+Char+CoVe (McCann et al., 2017) 53.7 91.8 Table 1: Test accuracy on SST-5 and IMDB.”
Interpretable Convolutional Neural Networks with Dual Local and Global Attention for Review Rating Prediction (2017, methodology): “Most networks use word embeddings [12, 22, 24, 25, 28] as the input layer, while [5, 7, 29] used character embeddings, especially [5] uses very deep (29 layers) CNNs on character embeddings and shows impressive results on text classi cation, including the Amazon reviews.”
Neural Sentiment Classification with User and Product Attention (2016, methodology): “(Li et al., 2015), (Tai et al., 2015) investigate tree-structured long-short term memory (LSTM) networks on text or sentiment classification.”
Supersense Embeddings: A Unified Model for Supersense Interpretation, Prediction, and Utilization (2016, background): “Both Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) are state-of-the-art semantic composition models for a variety of text classification tasks (Kim, 2014; Li et al., 2015; Johnson and Zhang, 2014).”
Learning Semantic Representations of Users and Products for Document Level Sentiment Classification (2015, methodology): “Li et al. (2015) compare the effectiveness of recursive neural network and recurrent neural network on five NLP tasks including sentiment classification.”
Neural Network Methods for Natural Language Processing (2017, unknown): “ese bidirectional models produce strong results for the task [Li et al., 2015].”