Very Deep Convolutional Networks for Text Classification

By Holger Schwenk, Loïc Barrault, Alexis Conneau, Yann LeCun. 2016

1,022 citations recorded by the source corpus.

The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which have pushed the state-of-the-art in computer vision. We present a new architecture (VDCNN) for text processing which operates directly at the character level and uses only small convolutions and pooling operations. We are able to show that the performance of this model increases with the depth: using up to 29 convolutional layers, we report improvements over the state-of-the-art on several public text classification tasks. To the best of our knowledge, this is the first time that very deep convolutional nets have been applied to text processing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Luke S. Zettlemoyer
4. Percy Liang
5. Yoshua Bengio
6. Graham Neubig
7. Quoc V. Le
8. Kyunghyun Cho

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yi Yang
3. Dacheng Tao
4. Xiaogang Wang
5. Trevor Darrell
6. Andrew Zisserman
7. Ming-Hsuan Yang
8. Cordelia Schmid

In Natural Language Processing

1. Jianfeng Gao
2. Mohit Bansal
3. Jason Aaron Edward Weston
4. Graham Neubig
5. Kyunghyun Cho
6. Luke S. Zettlemoyer
7. Yoav Goldberg
8. Erik Cambria

In Model Architectures

1. Yoshua Bengio
2. Vladlen Koltun
3. J. Kolter
4. Shaojie Bai
5. Kyunghyun Cho
6. Yoav Goldberg
7. Armand Joulin
8. Quoc V. Le

Citations in the corpus

• Bag of Tricks for Efficient Text Classification (2016, background): “Recently, models based on neural networks have become increasingly popular (Kim, 2014; Zhang and LeCun, 2015; Conneau et al., 2016).”
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, neutral): “These generic architectures can be applied to tasks like sentence classification (Zhang et al., 2015; Conneau et al., 2016) and sentence matching (Wan et al., 2016; He et al., 2016), but the model is trained only on data of a particular task.”
• Graph Convolutional Networks for Text Classification (2018, background): “(Zhang, Zhao, and LeCun 2015) and (Conneau et al. 2017) designed character level CNNs and achieved promising results.”
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “More recently, convolutional networks were applied to sentence classification (Kalchbrenner et al., 2014; Kim, 2014) and document classification (Zhang et al., 2015; Conneau et al., 2017; Johnson & Zhang, 2015; 2017).”
• Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, methodology): “, 2016) for image classification, the VDCNN network (Conneau et al., 2017) for text classification and Transformer-base (Vaswani et al.”
• Joint Embedding of Words and Labels for Text Classification (2018, methodology): “…( ii ) sophisticated deep CNN/RNN models: large/small word CNN, LSTM reported in (Zhang et al., 2015; Dai and Le, 2015) and deep CNN (29 layer) (Conneau et al., 2017); ( iii ) simple compositional methods: fastText (Joulin et al., 2016) and simple word embedding models (SWEM) (Shen et al.,…”
• HotFlip: White-Box Adversarial Examples for Text Classification (2017, background): “This classifier was able to outperform (Conneau et al., 2017), which has achieved the state-of-the-art result on some benchmarks, on AG’s news.”
• How to Fine-Tune BERT for Text Classification? (2019, neutral): “We compare our model with the following a variety of different methods: CNN-based methods such as Char-level CNN (Zhang et al., 2015), VD-CNN (Conneau et al., 2016) and DPCNN (Johnson and Zhang, 2017); RNN-based models such as D-LSTM (Yogatama et al., 2017), Skim-LSTM (Seo et al., 2017) and hierarch”
• StarSpace: Embed All The Things! (2017, background): “Several more recent works including (Tang, Qin, and Liu 2015), (Zhang and LeCun 2015), (Conneau et al. 2016), TagSpace (Weston, Chopra, and Adams 2014) and fastText (Joulin et al. 2016) have yielded good results on classification tasks such as sentiment analysis or hashtag prediction.”
• Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, methodology): “In future work, we aim to use more advanced CNN architectures (Conneau et al., 2016) for learning generic sentence embeddings.”
• Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, neutral): “text understanding [3] to Game AI [4].”
• HABERTOR: An Efficient and Effective Deep Hatespeech Detector (2020, methodology): “…our models with 15 state-ofthe-art baselines: Bag of Words (BOW) (Dinakar et al., 2011; Van Hee et al., 2015), NGRAM, CNN (Kim, 2014), VDCNN (Conneau et al., 2017), FastText (Joulin et al., 2016), LSTM (Cho et al., 2014), att-LSTM, RCNN (Lai et al., 2015), att-BiLSTM (Lin et al., 2017),…”
• A Survey of the Usages of Deep Learning for Natural Language Processing (2020, background): “[191] later showed that networks that employ a large number of convolutional layers work well for document classification.”
• Deep Learning--based Text Classification (2020, neutral): “[55] present a Very Deep CNN (VDCNN) model for text processing.”