Self-Adaptive Hierarchical Sentence Model

By Han Zhao, Zhengdong Lu, Pascal Poupart. 2015

191 citations recorded by the source corpus.

The ability to accurately model a sentence at varying stages (e.g., word-phrase-sentence) plays a central role in natural language processing. As an effort towards this goal we propose a self-adaptive hierarchical sentence model (AdaSent). AdaSent effectively forms a hierarchy of representations from words to phrases and then to sentences through recursive gated local composition of adjacent segments. We design a competitive mechanism (through gating networks) to allow the representations of the same sentence to be engaged in a particular learning task (e.g., classification), therefore effectively mitigating the gradient vanishing problem persistent in other recursive models. Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “, 2016), except for Naive Bayes - SVM and AdaSent results coming from (Zhao et al., 2015).”
Skip-Thought Vectors (2015, background): “In recent years, several approaches have been developed for learning composition operators that map word vectors to sentence vectors including recursive networks [1], recurrent networks [2], convolutional networks [3, 4] and recursive-convolutional methods [5, 6] among others.”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, neutral): “Results 1 correspond to AdaSent (Zhao et al., 2015), 2 to BLSTM-2DCNN (Zhou et al., 2016), 3 to TF-KLD (Ji and Eisenstein, 2013) and 4 to Illinois-LH system (Lai and Hockenmaier, 2014).”
Representation Learning with Contrastive Predictive Coding (2018, background): “We note that better [49, 27] results have been published on these target datasets, by transfer learning from a different source task.”
Towards Universal Paraphrastic Sentence Embeddings (2015, background): “…(Socher et al., 2011; 2012; 2013; ˙Irsoy & Cardie, 2014; Wieting et al., 2015), recursive networks based on non-syntactic hierarchical structure (Zhao et al., 2015; Chen et al., 2015b), convolutional neural networks (Kalchbrenner et al., 2014; Kim, 2014; Hu et al., 2014; Yin & Sch¨utze, 2015;…”
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (2018, methodology): “…the performance threshold Model MR SST-1 SST-2 Subj TREC RAE (Socher et al., 2011b) 77.7 43.2 82.4 – – MV-RNN (Socher et al., 2012) 79.0 44.4 82.9 – – LSTM (Tai et al., 2015) – 46.4 84.9 – – RNN (Zhao et al., 2015) 77 80% testing accuracy, SWEM exhibits a lower d int than CNN on AG News dataset.”
Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling (2016, neutral): “AdaSent: Self-adaptive hierarchical sentence model (Zhao et al., 2015).”
How Transferable are Neural Networks in NLP Applications? (2016, neutral): “9 +1:3 71.3 (RNN, Bowman ,2015) MSRP 69:0 0:5 69.6 (Arc-I CNN, Hu+,2014) Table 2: Accuracy (%) without transfer. We also include related models for comparison (Dong et al., 2015; Socher et al., 2011; Zhao et al., 2015; Bowman et al., 2015; Hu et al., 2014), showing that we have achieved comparable r”
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, neutral): “ 75.1/82.3 0.885 86.3 BiLSTM-Max (on AllNLI) (Conneau et al., 2017)y 81.1 86.3 92.4 90.2 84.6 88.2 76.2/83.1 0.884 86.3 Supervised (Direct) Naive Bayes - SVM 79.4 81.8 93.2 86.3 83.1 - - - - AdaSent (Zhao et al., 2015) 83.1 86.3 95.5 93.3 - 92.4 - - - TF-KLD (Ji and Eisenstein, 2013) - - - - - - 80.”
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding (2017, methodology): “Test Accu cBoW (Zhao, Lu, and Poupart 2015) 87.3 RNN (Zhao, Lu, and Poupart 2015) 90.2 CNN (Kim 2014) 93.6 AdaSent (Zhao, Lu, and Poupart 2015) 92.4 Skip-Thought (Kiros et al. 2015) 92.2 DiSAN 94.4 Table 5: Experimental results of different methods on TREC question-type classiﬁcation dataset.”
Learning Joint Multilingual Sentence Representations with Neural Machine Translation (2017, background): “(Zhao et al., 2015).”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, neutral): “(Zhao et al., 2015) 83.6 (1.6) 90.4 (0.7) 92.2 (1.2) 91.1 (1.0) / / SRU (Lei & Zhang, 2017) 84.8 (1.3) 89.7 (1.1) 93.4 (0.8) 93.9 (0.6) 89.1 (0.3) / Wide CNNs (Lei & Zhang, 2017) 82.2 (2.2) 88.8 (1.2) 92.9 (0.7) 93.2 (0.5) 85.3 (0.4) / Bi-LSTM (Graves et al., 2013) 84.6 (1.6) 90.2 (0.9) 94.7 (0.7)…”