Deep Pyramid Convolutional Neural Networks for Text Categorization

By Rie Johnson, Tong Zhang. 2017

772 citations recorded by the source corpus.

This paper proposes a low-complexity word-level deep convolutional neural network (CNN) architecture for text categorization that can efficiently represent long-range associations in text. In the literature, several deep and complex neural networks have been proposed for this task, assuming availability of relatively large amounts of training data. However, the associated computational complexity increases as the networks go deeper, which poses serious challenges in practical applications. Moreover, it was shown recently that shallow word-level CNNs are more accurate and much faster than the state-of-the-art very deep nets such as character-level CNNs even in the setting of large training data. Motivated by these findings, we carefully studied deepening of word-level CNNs to capture global representations of text, and found a simple network architecture with which the best accuracy can be obtained by increasing the network depth without increasing computational cost by much. We call it deep pyramid CNN. The proposed model with 15 weight layers outperforms the previous best models on six benchmark datasets for sentiment classification and topic categorization.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, methodology): “Pre-processing We use the same pre-processing as in earlier work (Johnson and Zhang, 2017; McCann et al., 2017).”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “Model IMDB Yelp-2 Yelp-5 DBpedia AG Amazon-2 Amazon-5 CNN [14] - 2.”
A Surprisingly Robust Trick for the Winograd Schema Challenge (2019, methodology): “For tasks with scarce data, transfer learning is often applied (Howard and Ruder, 2018; Johnson and Zhang, 2017), i.”
Fine-tuned Language Models for Text Classification (2018, methodology): “Datasets and tasks We evaluate our method on ﬁve widely-studied datasets of different sizes used by state-of-the-art text classiﬁcation and transfer learning approaches (Johnson and Zhang, 2017; McCann et al., 2017) sis, question classiﬁcation, and topic classiﬁcation.”
Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “Several prior works have suggested that either complex pretraining schemes using unsupervised methods such as language modeling (Dai and Le 2015; Miyato, Dai, and Goodfellow 2016) or complicated models (Johnson and Zhang 2017) are necessary to achieve a high classification accuracy.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “More recently, convolutional networks were applied to sentence classiﬁcation (Kalchbrenner et al., 2014; Kim, 2014) and document classiﬁcation (Zhang et al., 2015; Conneau et al., 2017; Johnson & Zhang, 2015; 2017).”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “We compare our model with the following a variety of different methods: CNN-based methods such as Char-level CNN (Zhang et al., 2015), VD-CNN (Conneau et al., 2016) and DPCNN (Johnson and Zhang, 2017); RNN-based models such as D-LSTM (Yogatama et al., 2017), Skim-LSTM (Seo et al., 2017) and hierarch”
A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances (2022, methodology): “In 2020, Zhang et al. [20] introduced different feature extraction, feature reduction, and ML-based classifiers in terms of the standard pipeline for multi-channel EEG emotion recognition, and discussed the recent advances of multimodal emotion recognition based on conventional ML or DL techniques.”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, neutral): “CNN [103] 91.33 Deep Pyramid CNN [104] 93.13 CNN [105] 93.43 Universal Language Model Fine-tuning (ULMFiT) [106] 94.99”
Potential, challenges and future directions for deep learning in prognostics and health management applications (2020, background): “Classic supervised tasks in other fields include image classification [80, 81], and text categorization [82, 83, 84].”
Unsupervised Data Augmentation (2019, methodology): “In the fully supervised settings, the preBERT SOTAs include ULMFiT [Howard and Ruder, 2018] for Yelp-2 and Yelp-5, DPCNN [Johnson and Zhang, 2017] for Amazon-2 and Amazon-5, Mixed VAT [Sachan et al.”
Deep Learning--based Text Classification (2020, methodology): “In References [48, 49], instead of using pre-trained lowdimensional word vectors as input to CNNs, the authors directly apply CNNs to high-dimensional text data to learn the embeddings of small text regions for classification.”
How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence (2020, methodology): “We have implemented several classical encoding models in NLP for element extraction, including TextCNN (Kim, 2014), DPCNN (Johnson and Zhang, 2017), LSTM (Hochreiter and Schmidhuber, 1997), BiDAF (Seo et al.”
Improving the Interpretability of Neural Sentiment Classifiers via Data Augmentation (2019, background): “, 2013; Pang et al., 2002; Liu, 2012). The recent development of neural network modeling has largely boosted the prediction performance (e.g., accuracy) on sentiment classiﬁcation (Yang et al., 2016; Johnson and Zhang, 2017, 2016; Zhou et al., 2016), while the nonlinearity of neural network models h”
A Survey on Text Classification: From Traditional to Deep Learning (2020, methodology): “The SICK contains almost 10,000 English sentence pairs.”
JEC-QA: A Legal-Domain Question Answering Dataset (2019, methodology): “There are 15 topics in total, and we employ 3 representative models, including BERT (Devlin et al. 2018), TextCNN (Kim 2014), and DPCNN (Johnson and 1 https://www.elastic.co/ Zhang 2017).”