Neural Tree Indexers for Text Understanding

By Tsendsuren Munkhdalai, Hong Yu. 2016

106 citations recorded by the source corpus.

Recurrent neural networks (RNNs) process input text sequentially and model the conditional transition between word tokens. In contrast, the advantages of recursive networks include that they explicitly model the compositionality and the recursive structure of natural language. However, the current recursive architecture is limited by its dependence on syntactic tree. In this paper, we introduce a robust syntactic parsing-independent tree structured model, Neural Tree Indexers (NTI) that provides a middle ground between the sequential RNNs and the syntactic treebased recursive models. NTI constructs a full n-ary tree by processing the input text with its node function in a bottom-up fashion. Attention mechanism can then be applied to both structure and node function. We implemented and evaluated a binary tree model of NTI, showing the model achieved the state-of-the-art performance on three different NLP tasks: natural language inference, answer sentence selection, and sentence classification, outperforming state-of-the-art recurrent and recursive neural networks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, methodology): “…2016b) 83.3% 300D Tree-based CNN encoders (Mou et al., 2015a) 82.1% 300D SPINN-PI encoders (Bowman et al., 2016) 83.2% 300D NTI-SLSTM-LSTM encoders (Munkhdalai & Yu, 2016a) 83.4% 1024D GRU encoders with SkipThoughts pre-training (Vendrov et al., 2015) 81.4% 300D NSE encoders (Munkhdalai & Yu,…”
Structured Attention Networks (2017, neutral): “2 Neural Tree Indexers (Munkhdalai & Yu, 2016) 87 .”
Learned in Translation: Contextualized Word Vectors (2017, neutral): “…in NLP. [Looks et al., 2017] 89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8…”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Sentiment Analysis Because SST came with parse trees for every example, some approaches use all of the sub-tree labels by modeling trees explicitly [Yu and Munkhdalai, 2017b, Tai et al., 2015] as in the original paper.”
Enhanced LSTM for Natural Language Inference (2016, methodology): “The model of Munkhdalai and Yu (2016b) extends the framework of Wang and Jiang (2016) to a full n-ary tree model and achieves further improvement.”
Recurrent Neural Networks for Classifying Relations in Clinical Notes (2017, neutral): “In general domain NLP, a growing number of studies have successfully used recurrent neural networks (RNNs) combined with word embedding [22] on tasks including language modeling [23], text classification [24–27], question answering [25,26,28,29], machine translation [25,30–32], named entity recognit”
Neural Semantic Encoders (2016, neutral): “8 Full tree matching NTI-SLSTM-LSTM global attention (Munkhdalai and Yu, 2017) 300 3.”
Bilateral Multi-Perspective Matching for Natural Language Sentences (2017, neutral): “…al. , 2015] 81.4 [Mou et al. , 2015] 82.1 [Rockt¨aschel et al. , 2015] 83.5 [Liu et al. , 2016b] 85.0 [Liu et al. , 2016a] 85.1 [Wang and Jiang, 2015] 86.1 [Cheng et al. , 2016] 86.3 [Parikh et al. , 2016] 86.8 [Munkhdalai and Yu, 2016] 87.3 [Sha et al. , 2016] 87.5 [Chen et al. , 2016] tion (Eq.”
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding (2017, neutral): “ree-based CNN encoders (Mou et al. 2016) 3.5m 83.3 82.1 300D SPINN-PI encoders (Bowman et al. 2016) 3.7m 89.2 83.2 600D Bi-LSTM encoders (Liu et al. 2016) 2.0m 86.4 83.3 300D NTI-SLSTM-LSTM encoders (Munkhdalai and Yu 2017b) 4.0m 82.5 83.4 600D Bi-LSTM encoders+intra-attention (Liu et al. 2016) 2.8m”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, neutral): “…89.2 83.2 600D Bi-LSTM encoders (Liu et al., 2016) 2.0m 86.4 83.3 300D NTI-SLSTM-LSTM encoders (Munkhdalai & Yu, 2017b) 4.0m 82.5 83.4 600D Bi-LSTM encoders+intra-attention (Liu et al., 2016) 2.8m 84.5 84.2 300D NSE encoders (Munkhdalai & Yu, 2017a) 3.0m 86.2 84.6 600D (300+300) Deep Gated Attn.”
Do latent tree learning models identify meaningful structure in sentences? (2017, methodology): “Several models (Naradowsky et al., 2012; Kim et al., 2017; Liu and Lapata, 2018; Munkhdalai and Yu, 2017a) have also been proposed that can induce latent dependency trees over text using mechanisms like attention, but do not propagate information up the trees as in typical compositional models.”
Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference (2016, methodology): “The original SNLI corpus contains also “ the other ” category, which includes the sentence pairs lacking consensus among multiple human annotators.”
Convolutional Interaction Network for Natural Language Inference (2018, unknown): “…Match-LSTM (Wang and Jiang, 2016) 92.0 86.1 Decomposable attention model (Parikh et al., 2016) 90.5 86.8 BiMPM (Zhiguo Wang, 2017) 90.9 87.5 NTI-SLSTM-LSTM (Munkhdalai and Yu, 2017) 88.5 87.3 Re-read LSTM (Sha et al., 2016) 90.7 87.5 DIIN (Gong et al., 2017) 91.2 88.0 ESIM (Chen et al., 2017a) 92”
Enhancing Sentence Embedding with Generalized Pooling (2018, unknown): “3 300D NTI-SLSTM-LSTM (Yu and Munkhdalai, 2017b) 83.”
Neural Natural Language Inference Models Enhanced with External Knowledge (2017, unknown): “8 NTI (Yu and Munkhdalai, 2017b) 87.”
DR-BiLSTM: Dependent Reading Bidirectional LSTM for Natural Language Inference (2018, unknown): “8% (Yu and Munkhdalai, 2017b) 88.”
Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference (2017, background): “The previous state-of-the-art sentence encoder-based model (Munkhdalai and Yu, 2016b), called neural semantic encoders (NSE), only achieved an accuracy of 84.”
Natural Language Inference with External Knowledge (2017, methodology): “…models, which often need relatively large amounts of annotated data to estimate their parameters, have shown to achieve the state of the art on SNLI and MultiNLI (Bowman et al., 2015; 2016; Munkhdalai & Yu, 2016b; Parikh et al., 2016; Sha et al., 2016; Paria et al., 2016; Chen et al., 2017a;b).”