Molding CNNs for text: non-linear, non-consecutive convolutions

By Tao Lei, Regina Barzilay, Tommi S. Jaakkola. 2015

148 citations recorded by the source corpus.

The success of deep learning often derives from well-chosen operational building blocks. In this work, we revise the temporal convolution operation in CNNs to better adapt it to text processing. Instead of concatenating word representations, we appeal to tensor algebra and use low-rank n-gram tensors to directly exploit interactions between words already at the convolution stage. Moreover, we extend the n-gram convolution to non-consecutive words to recognize patterns with intervening words. Through a combination of low-rank tensors, and pattern weighting, we can efficiently evaluate the resulting convolution operation via dynamic programming. We test the resulting architecture on standard sentiment classification and news categorization tasks. Our model achieves state-of-the-art performance both in terms of accuracy and training speed. For instance, we obtain 51.2% accuracy on the fine-grained sentiment classification task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Long Short-Term Memory-Networks for Machine Reading (2016, methodology): “…(Socher et al., 2013) 45.7 85.4 DRNN (Irsoy and Cardie, 2014) 49.8 86.6 DCNN (Blunsom et al., 2014) 48.5 86.8 CNN-MC (Kim, 2014) 48.0 88.1 T-CNN (Lei et al., 2015) 51.2 88.6 PV (Le and Mikolov, 2014) 48.7 87.8 CT-LSTM (Tai et al., 2015) 51.0 88.0 LSTM (Tai et al., 2015) 46.4 84.9 2-layer LSTM…”
Rationalizing Neural Predictions (2016, methodology): “We use the “one-versus-all” hinge loss (i.e. positive versus other negatives) for the encoder, similar to (Lei et al., 2015b).”
Character-Aware Neural Language Models (2015, background): “to be been successful for sentence/document classification (Kalchbrenner et al. 2014; Kim 2014; Zhang and LeCun 2015; Lei et al. 2015), and we posit that further gains could be achieved by employing highway layers on top of existing CNN architectures.”
Ranking Sentences for Extractive Summarization with Reinforcement Learning (2018, neutral): “In recent years, CNNs have proven useful for various NLP tasks (Collobert et al., 2011; Kim, 2014; Kalchbrenner et al., 2014; Zhang et al., 2015; Lei et al., 2015; Kim et al., 2016; Cheng and Lapata, 2016) because of their effectiveness in identifying salient patterns in the input (Xu et al.”
Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling (2016, neutral): “Molding-CNN: Molding CNNs for text: non-linear, non-consecutive convolutions (Lei et al., 2015).”
Semi-supervised Question Retrieval with Gated Convolutions (2015, methodology): “Our approach is inspired by temporal convolutional neural networks (LeCun et al., 1998) and, in particular, its recent reﬁne-ment (Lei et al., 2015), tailored to capture longer-range, non-consecutive patterns in a weighted manner.”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, neutral): “It is mainly used for sentence-encoding tasks (Lei et al., 2015; Kalchbrenner et al., 2014).”
A C-LSTM Neural Network for Text Classification (2015, methodology): “In a more recent work (Lei et al., 2015), Tao et al. apply tensor-based operations between words to replace linear operations on concatenated word vectors in the standard convolutional layer and explore the non-linear interactions between nonconsective n-grams.”
Neural Extractive Summarization with Side Information (2017, background): “, 2012) and for various NLP tasks (Collobert et al., 2011; Kim, 2014; Kalchbrenner et al., 2014; Zhang et al., 2015; Lei et al., 2015; Kim et al., 2016; Cheng and Lapata, 2016).”
Training RNNs as Fast as CNNs (2017, methodology): “SRU can be viewed as a simpliﬁed version (or special case) of Quasi-RNN, RCNN and KNN with window size 1 and additional highway connections.”
Bridging CNNs, RNNs, and Weighted Finite-State Machines (2018, methodology): “Similarly to SoPa, several recently introduced variants of CNNs support varying window sizes by either allowing several ﬁxed window sizes (Yin and Sch¨utze, 2015) or by supporting non-consecutive n -gram matching (Lei et al., 2015; Nguyen and Grishman, 2016).”
Combining Neural Networks and Log-linear Models to Improve Relation Extraction (2015, background): “e, the k-grams) of the relation mentions [Nguyen and Grishman, 2015a; Lei et al., 2015] while the latter adaptively accumulate the context information in the whole sentence via the memory units, thereby encoding the long and possibly non-consecutive patterns for RE [Hochreiter and Schmidhuber, 1997]”
Star-Transformer (2019, unknown): “BiLSTM (Li et al., 2015) 49.8 Tree-LSTM (Tai et al., 2015) 51.0 CNN-Tensor (Lei et al., 2015) 51.2 Emb + self-att (Shen et al., 2018a) 48.9 BiLSTM + self-att (Yoon et al., 2018) 50.4 CNN + self-att (Yoon et al., 2018) 50.6 Dynamic self-att (Yoon et al., 2018) 50.6 DiSAN (Shen et al., 2018a) 51.7…”
Fast and Accurate Entity Recognition with Iterated Dilated Convolutions (2017, background): “Lei et al. (2015) present a CNN variant where convolutions adaptively skip neighboring words.”
Denoising Bodies to Titles: Retrieving Similar Questions with Recurrent Convolutional Models (2015, methodology): “In particular, we incorporate adaptive gating in non-consecutive CNNs (Lei et al., 2015) so as to focus temporal averaging in these models on key pieces of the questions.”
A Deep Network with Visual Text Composition Behavior (2017, methodology): “(Lei et al., 2015).”
Application of Generative Autoencoder in De Novo Molecular Design (2017, background): “[21] A CNN consists of an input and output layer as well as multiple hidden layers.”