Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

By Kai Sheng Tai, Richard Socher, Christopher D. Manning. 2015

3,266 citations recorded by the source corpus.

Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree-LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank).

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “Additional works have also been done in exploiting linguistic structures such as parse and dependence trees to improve sentence representations (Ma et al., 2015; Mou et al., 2015b; Tai et al., 2015).”
Recurrent Neural Network Grammars (2016, background): “…neural architectures have been explored in a number of applications, including discriminative parsing (Socher et al., 2013a; Kiperwasser and Goldberg, 2016), sentiment analysis (Tai et al., 2015; Socher et al., 2013b), and sentence representation (Socher et al., 2011; Bowman et al., 2006).”
An efficient framework for learning sentence representations (2018, methodology): “The models are AdaSent (Zhao et al., 2015), CNN (Kim, 2014), TF-KLD (Ji & Eisenstein, 2013) and Dependency-Tree LSTM (Tai et al., 2015).”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “We obtain a pear-son score of 0.885 on SICK-R while (Tai et al., 2015) obtained 0.868, and we obtain 86.3% test accuracy on SICK-E while previous best hand-engineered models (Lai and Hockenmaier, 2014) obtained 84.5%.”
Skip-Thought Vectors (2015, methodology): “Only the dependency tree-LSTM of [22] performs better than our results.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Sentiment Analysis Because SST came with parse trees for every example, some approaches use all of the sub-tree labels by modeling trees explicitly [Yu and Munkhdalai, 2017b, Tai et al., 2015] as in the original paper.”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, methodology): “Lattice is encoded with a variant of TreeLSTM (Tai et al., 2015), which requires changing the model architecture.”
Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “Previous work in (Tai et al., 2015; Wang and Jiang, 2016) shows this method can build better matching representation.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “, 2013], and later, Tree-LSTMs [Tai et al., 2015].”
Language Understanding for Text-based Games using Deep Reinforcement Learning (2015, methodology): “, 2014) and sentiment analysis (Tai et al., 2015) to compose vector representations of sentences from word-level embeddings (Mikolov et al.”
Gated Graph Sequence Neural Networks (2015, methodology): “Similarly, Recursive Neural Networks (Goller & Kuchler, 1996; Socher et al., 2011) being extended to Tree LSTMs (Tai et al., 2015) is analogous to our using of GRU updates in GG-NNs instead of the standard GNN recurrence with the aim of improving the long-term propagation of information across a gra”
Convolutional Networks on Graphs for Learning Molecular Fingerprints (2015, background): “Techniques from natural language processing [25] might be fruitfully adapted to this domain.”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “There are two representative approaches: building structured neural networks with explicit phrase-based structures or segments [37, 51, 41, 34]; and using data augmentation techniques [3, 17, 2].”
Probing for semantic evidence of composition by means of simple classification tasks (2016, methodology): “SICK is one of the most commonly used benchmark tasks for evaluating composition models (Kiros et al., 2015; Tai et al., 2015; Wiet-ing et al., 2015).”
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015, background): “We suspect that this is due to the recurrent input sequence model having trouble modeling very long inputs.”
Enhanced LSTM for Natural Language Inference (2016, methodology): “trees of a premise and hypothesis through treeLSTM (Zhu et al., 2015; Tai et al., 2015; Le and Zuidema, 2015), which extends the chain LSTM to a recursive network (Socher et al.”
On Calibration of Modern Neural Networks (2017, methodology): “On SST, we train TreeLSTMs (Long Short Term Memory) (Tai et al., 2015).”