Natural Language Inference by Tree-Based Convolution and Heuristic Matching

By Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, Zhi Jin. 2015

368 citations recorded by the source corpus.

In this paper, we propose the TBCNN-pair model to recognize entailment and contradiction between two sentences. In our model, a tree-based convolutional neural network (TBCNN) captures sentence-level semantics; then heuristic matching layers like concatenation, element-wise product/difference combine the information in individual sentences. Experimental results show that our model outperforms existing sentence encoding-based approaches by a large margin.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “Additional works have also been done in exploiting linguistic structures such as parse and dependence trees to improve sentence representations (Ma et al., 2015; Mou et al., 2015b; Tai et al., 2015).”
Structured Attention Networks (2017, neutral): “6 Tree-Based CNN (Mou et al., 2016) 82 .”
Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, background): “Recently, a heuristic matching trick with difference and element-wise product is found effective in combining different representations (Mou et al., 2016; Chen et al., 2017b): (10) where ◦ denotes the element-wise product, and W f , b f are trainable parameters.”
A Decomposable Attention Model for Natural Language Inference (2016, neutral): “0M 300D tree-based CNN encoders (Mou et al., 2015) 83.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “• Sentence-pair tasks: We perform the same steps on both sentences and use the heuristic feature vector [h1;h2;h1 · h2;h1 − h2] in the MLP, following Mou et al. (2016). When training target-task models on QQP, STS, MNLI, and QNLI, we use a cross-sentence attention mechanism similar to BiDAF (Seo et ”
Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (2018, methodology): “All models predict the label using a concatenation of features derived from the sentence representations (e.g. maximum, mean), for example as in Mou et al. (2016).”
Enhanced LSTM for Natural Language Inference (2016, methodology): “4 (4) 300D tree-based CNN encoders (Mou et al., 2016) 3.”
SciTaiL: A Textual Entailment Dataset from Science Question Answering (2018, methodology): “Along with their vector representations, we use element-wise difference and products (Mou et al. 2016) as inputs to this linear perceptron.”
A Fast Unified Model for Parsing and Sentence Understanding (2016, methodology): “This feature vector consists of the concatenation of these two sentence vectors, their difference, and their elementwise product (following Mou et al., 2016): (6) ~xclassifier = ~hpremise ~hhypothesis ~hpremise − ~hhypothesis ~hpremise ~hhypothesis This feature vector is then passed to a series…”
Natural Language Inference over Interaction Space (2017, methodology): “The convolution structure is also applied on different level of granularity such as byte(Zhang & LeCun, 2017), character(Zhang et al., 2015), word(Gehring et al., 2017) and sentences(Mou et al., 2015) levels.”
Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment (2018, methodology): “…important, because the SNLI corpus is widely used as training data by many NN models for RTE (Bowman et al., 2015; Rocktäschel et al., 2015; Yin et al., 2016; Mou et al., 2016a; Wang and Jiang, 2016; Liu et al., 2016a; Liu et al., 2016b; Cheng et al., 2016; Parikh et al., 2016; Sha et al., 2016).”
Neural Tree Indexers for Text Understanding (2016, methodology): “Unless otherwise noted, we follow the setting in the previous work (Mou et al., 2016; Bowman et al., 2016) and use an MLP for classification which takes in NTI outputs and computes the concatenation [hp2n−1;h h 2n−1], absolute difference hp2n−1 − h2n−1 and elementwise product hp2n−1 ·h2n−1 of the tw”
Shortcut-Stacked Sentence Encoders for Multi-Domain Inference (2017, methodology): “…proposed models can be categorized into two groups: (1) encoding-based models (or sentence encoders), such as Tree-based CNN encoders (TBCNN) in Mou et al. (2015) or Stack-augmented Parser-Interpreter Neural Network (SPINN) in Bowman et al. (2016), and (2) joint, pairwise models that use…”
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, methodology): “We use the heuristic matching feature vector [h1;h2;h1·h2; |h1−h2|] (Mou et al., 2016) as input to an MLP.”
Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention (2016, methodology): “• Concatenation of the two representations • Element-wise product • Element-wise difference This matching architecture was ﬁrst used by (Mou et al., 2015).”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “For smaller sentence pair-tasks, we perform these steps on both sentences and use the heuristic matching feature vector [ h 1 ; h 2 ; h 1 · h 2 ; h 1 − h 2 ] in the MLP, following Mou et al. (2016).”
Deconvolutional Latent-Variable Model for Text Sequence Matching (2017, background): “each sentence is mapped to a vector (embedding), and two such vectors are used for predictions of relationships between the corresponding two sentences (Bowman et al., 2016a; Mou et al., 2015).”