Discriminative Neural Sentence Modeling by Tree-Based Convolution

By Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, Zhi Jin. 2015

127 citations recorded by the source corpus.

This paper proposes a tree-based convolutional neural network (TBCNN) for discriminative sentence modeling. Our models leverage either constituency trees or dependency trees of sentences. The tree-based convolution process extracts sentences' structural features, and these features are aggregated by max pooling. Such architecture allows short propagation paths between the output layer and underlying feature detectors, which enables effective structural feature learning and extraction. We evaluate our models on two tasks: sentiment analysis and question classification. In both experiments, TBCNN outperforms previous state-of-the-art results, including existing neural networks and dedicated feature/rule engineering. We also make efforts to visualize the tree-based convolution process, shedding light on how our models work.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “Additional works have also been done in exploiting linguistic structures such as parse and dependence trees to improve sentence representations (Ma et al., 2015; Mou et al., 2015b; Tai et al., 2015).”
Universal Language Model Fine-tuning for Text Classification (2018, neutral): “(Johnson and Zhang, 2016) 5.9 TBCNN (Mou et al., 2015) 4.0 Virtual (Miyato et al., 2016) 5.9 LSTM-CNN (Zhou et al., 2016) Johnson and Zhang (2017).”
Learned in Translation: Contextualized Word Vectors (2017, neutral): “…89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8 LSTM-CNN [Zhou et al., 2016] 96.1”
Fine-tuned Language Models for Text Classification (2018, methodology): “st CoVe (McCann et al., 2017), a stateModel Test Model Test IMDb BCN+Char+CoVe (McCann et al., 2017) 91.8 TREC-6 BCN+Char+CoVe (McCann et al., 2017) 95.8 oh-LSTM (Johnson and Zhang, 2016) 94.1 TBCNN (Mou et al., 2015) 96.0 Virtual (Miyato et al., 2016) 94.1 LSTM-CNN(Zhou et al., 2016) 96.1 FitLaM (O”
Natural Language Inference by Tree-Based Convolution and Heuristic Matching (2015, methodology): “The tree-based convolutoinal neural network (TBCNN) is ﬁrst proposed in our previous work (Mou et al., 2016) 2 to classify program source code; later, we further propose TBCNN variants to model sentences (Mou et al., 2015).”
Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling (2017, methodology): “An approach of Mou et al. (2015) to sentiment analysis and question classiﬁcation, introduced even before GCNs became popular in the machine learning community, is related to graph convolution.”
Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling (2016, methodology): “TBCNN: Discriminative neural sentence modeling by tree-based convolution (Mou et al., 2015).”
How Transferable are Neural Networks in NLP Applications? (2016, methodology): “2007; Daume III et al., 2010), instance weighting´ (Jiang and Zhai, 2007; Foster et al., 2010), struc1We release our code on https://sites.google.com/site/ transfernlp/, which is implemented based on Mou et al. (2015a). 2In this paper, we do not distinguish the conceptual difference between transfer”
Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths (2015, methodology): “Our implementation is built upon Mou et al. (2015).”
Convolutional Neural Networks over Tree Structures for Programming Language Processing (2014, methodology): “Regarding the proposed TBCNN, we extend it to process syntactic parse trees of natural languages (Mou et al. 2015); Duvenaud et al. (2015) apply a similar convolutional network over graphs to analyze molecules.”
A C-LSTM Neural Network for Text Classification (2015, methodology): “Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have emerged as two widely used architectures and are often combined with sequence-based or tree-structured models (Tai et al., 2015; Lei et al., 2015; Tang et al., 2015; Kim, 2014; Kalchbrenner et al., 2014; Mou et al., 2015)”
Aspect-Level Sentiment Analysis Via Convolution over Dependency Tree (2019, background): “(Mou et al., 2015) exploit the short paths of dependency trees to learn representations of sentences using convolutional neural networks, while preserving dependency information.”
Graph Convolutional Networks With Argument-Aware Pooling for Event Detection (2018, methodology): “Syntactic information has also been employed in neural network models for various natural language processing tasks, including sentiment analysis (Socher et al. 2013; Mou et al. 2015), dependency parsing (Le and Zuidema 2014; Dyer et al.”
Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation (2016, background): “In topological models, sentence representations are composed following given topological structures over words (Socher et al. 2011; Hermann and Blunsom 2013; Iyyer et al. 2014; Mou et al. 2015; Tai, Socher, and Manning 2015; Le and Zuidema 2015).”
Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System (2016, background): “Mou et al. [21, 20] propose syntax-aware convolution based on parse trees.”
Combination of Convolutional and Recurrent Neural Network for Sentiment Analysis of Short Texts (2016, unknown): “8 d-TBCNN(Mou et al., 2015) – 51.”
Rumor Detection on Twitter with Tree-structured Recursive Neural Networks (2018, methodology): “Mou et al. (2015) used a convolutional network over tree structures for syntactic tree parsing of natural language sentences.”
Distilling Word Embeddings: An Encoding Approach (2015, methodology): “The learned embeddings can be fed to standard neural networks for supervised learning, e.g., part-of-speech tagging (Collobert and We-ston, 2008), relation extraction (Zeng et al., 2014), sentiment analysis (Mou et al., 2015), etc.”
Neural Attribute Machines for Program Generation (2017, background): “While other attempts at learning models that can be used to generate trees include performing convolutions over nodes in binary trees [12] or stacking multiple RNNs in fixed directions [13], a natural paradigm to adopt for presenting a tree to a neural model is a single-traversal, top-down, left-to-”