Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

By Rie Johnson, Tong Zhang. 2014

910 citations recorded by the source corpus.

Convolutional neural network (CNN) is a neural network that can make use of the internal structure of data such as the 2D structure of image data. This paper studies CNN on text categorization to exploit the 1D structure (namely, word order) of text data for accurate prediction. Instead of using low-dimensional word vectors as input as is often done, we directly apply CNN to high-dimensional text data, which leads to directly learning embedding of small text regions for use in classification. In addition to a straightforward adaptation of CNN from image to text, a simple but new variation which employs bag-of-word conversion in the convolution layer is proposed. An extension to combine multiple convolution layers is also explored for higher accuracy. The experiments demonstrate the effectiveness of our approach in comparison with state-of-the-art methods.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Semi-supervised Sequence Learning (2015, neutral): “78% seq2-bown-CNN (ConvNet with dynamic pooling) [11] 7.”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “training one-hot CNN (JZ15a; Johnson and Zhang 2015a) or one-hot bidirectional LSTM (BiLSTM) network with dynamic max-pooling (JZ16; Johnson and Zhang 2016).”
Deep Pyramid Convolutional Neural Networks for Text Categorization (2017, methodology): “Note that ShallowCNN enhanced with unsupervised embeddings (row 2) was originally proposed in (Johnson and Zhang, 2015b) as a semi-supervised extension of (Johnson and Zhang, 2015a), and then it was tested on the large datasets in (Johnson and Zhang, 2016).”
Adversarial Training Methods for Semi-Supervised Text Classification (2016, methodology): “Adversarial and virtual adversarial training resemble some semi-supervised or transductive SVM approaches (Joachims, 1999; Chapelle & Zien, 2005; Collobert et al., 2006; Belkin et al., 2006) in that both families of methods push the decision boundary far from training examples (or in the case of tra”
Character-level Convolutional Networks for Text Classification (2015, background): “It has been shown that ConvNets can be directly applied to distributed [6] [16] or discrete [13] embedding of words, without any knowledge on the syntactic or semantic structures of a language.”
Hierarchical Attention Networks for Document Classification (2016, background): “Although neural-network–based approaches to text classiﬁcation have been quite effective (Kim, 2014; Zhang et al., 2015; Johnson and Zhang, 2014; Tang et al., 2015), in this paper we test the hypothesis that better representations can be obtained by incorporating knowledge of document structure in…”
Text Understanding from Scratch (2015, background): “This evidence is in contrast with various previous approaches where a dictionary of words is a necessary starting point, and usually structured parsing is hard-wired into the model(Collobert et al., 2011b)(Kim, 2014)(Johnson & Zhang, 2014)(dos Santos & Gatti, 2014).”
A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification (2015, background): “performance on the practically important task of sentence classification (Kim, 2014; Kalchbrenner et al., 2014; Johnson and Zhang, 2014; Zhang et al., 2016).”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “More recently, convolutional networks were applied to sentence classiﬁcation (Kalchbrenner et al., 2014; Kim, 2014) and document classiﬁcation (Zhang et al., 2015; Conneau et al., 2017; Johnson & Zhang, 2015; 2017).”
Document Modeling with Gated Recurrent Neural Network for Sentiment Classification (2015, methodology): “CNN and LSTM are stateof-the-art semantic composition models for sentiment classification (Kim, 2014; Kalchbrenner et al., 2014; Johnson and Zhang, 2015; Li et al., 2015a).”
Adversarial Feature Matching for Text Generation (2017, methodology): “There are other CNN architectures in the literature (Kalch-brenner et al., 2014; Hu et al., 2014; Johnson & Zhang, 2015).”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “For work using neural network architecture with sparse vector encodings see (Johnson & Zhang, 2015).”
Understanding Convolutional Neural Networks for Text Classification (2018, background): “…10k evenly split short (sentences or snippets) movie reviews. b) Elec : electronic product reviews for sentiment classiﬁcation introduced by Johnson and Zhang (2015), assembled from the Amazon review dataset (McAuley and Leskovec, 2013; McAuley et al., 2015 For word embeddings, we use the…”
Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings (2016, background): “In particular, let us ﬁrst focus on one-hot CNN which we proposed in JZ15 (Johnson & Zhang, 2015a;b).”
Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, background): “…on complex nonlinear functions like recursive neural networks (Socher et al., 2011, 2013), convolutional neural networks (Kalchbrenner et al., 2014; Hu et al., 2014; Johnson and Zhang, 2015; Zhang et al., 2015; Gan et al., 2017), and recurrent neural networks (Tai et al., 2015; Lin et al., 2017).”
Joint Deep Modeling of Users and Items Using Reviews for Recommendation (2017, methodology): “In this way, convolution operations make use of the internal structure of data and provide a mechanism for efficient use of words’ order in text modeling [11].”
Deconvolutional Paragraph Representation Learning (2017, background): “Previous work has considered leveraging CNNs as encoders for various natural language processing tasks [22, 28, 21, 38, 39].”