Very Deep Convolutional Networks for Natural Language Processing

By Alexis Conneau, Holger Schwenk, Loïc Barrault, Yann LeCun. 2016

323 citations recorded by the source corpus.

The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which are very successful in computer vision. We present a new architecture for text processing which operates directly on the character level and uses only small convolutions and pooling operations. We are able to show that the performance of this model increases with the depth: using up to 29 convolutional layers, we report significant improvements over the state-of-the-art on several public text classification tasks. To the best of our knowledge, this is the first time that very deep convolutional nets have been applied to NLP.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Deep Pyramid Convolutional Neural Networks for Text Categorization (2017, background): “For example, in (Conneau et al., 2016), very deep 32-layer character-level CNNs were shown to outperform deep 9-layer character-level CNNs of (Zhang et al.”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, background): “Deep convolutional neural networks (DCNNs) have achieved great success on many tasks across a variety of domains, such as vision [37, 60, 24, 23, 7], audition [26, 21, 11, 47], and natural language processing [68, 29, 9, 38].”
Bag of Tricks for Efficient Text Classification (2016, background): “Recently, models based on neural networks have become increasingly popular (Kim, 2014; Zhang and LeCun, 2015; Conneau et al., 2016).”
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, neutral): “These generic architectures can be applied to tasks like sentence classiﬁcation (Zhang et al., 2015; Conneau et al., 2016) and sentence matching (Wan et al., 2016; He et al., 2016), but the model is trained only on data of a particular task.”
DKN: Deep Knowledge-Aware Network for News Recommendation (2018, background): “Recently, inspired by the success of applying convolutional neural networks (CNN) in the filed of computer vision [23], researchers have proposed many CNN-based models for sentence representation learning [7, 19, 20, 53] 7 .”
Recent advances in convolutional neural networks (2015, background): “depended on the input feature map size. The CNN 27 architectures mentioned above are rather shallow compared with the deep CNNs which are very successful in computer vision. Recently, Conneau et al . [315] implement a deep convolutional architecture which is up to 29 convolutional layers. They nd th”
Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms (2018, methodology): “However, Deep CNN text models have also been developed (Conneau et al., 2016), and are considered in a few of our experiments.”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “We compare our model with the following a variety of different methods: CNN-based methods such as Char-level CNN (Zhang et al., 2015), VD-CNN (Conneau et al., 2016) and DPCNN (Johnson and Zhang, 2017); RNN-based models such as D-LSTM (Yogatama et al., 2017), Skim-LSTM (Seo et al., 2017) and hierarch”
Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, methodology): “In future work, we aim to use more advanced CNN architectures (Conneau et al., 2016) for learning”
Attention-Based Prototypical Learning Towards Interpretable, Confident and Robust Deep Neural Networks (2019, methodology): “Then, 8 convolution blocks as in (Conneau et al., 2016) are applied, with 64, 64, 128, 128, 256, 256, 512 and 512 filters respectively.”
Hash Embeddings for Efficient Word Representations (2017, methodology): “Another approach employed by Zhang et al. (2015); Xiao and Cho (2016); Conneau et al. (2016) is to use inputs that represent sub-word units such as syllables or individual characters rather than words.”
Interpretable Convolutional Neural Networks with Dual Local and Global Attention for Review Rating Prediction (2017, methodology): “Most networks use word embeddings [12, 22, 24, 25, 28] as the input layer, while [5, 7, 29] used character embeddings, especially [5] uses very deep (29 layers) CNNs on character embeddings and shows impressive results on text classi cation, including the Amazon reviews.”
Learning Longer-term Dependencies in RNNs with Auxiliary Losses (2018, unknown): “ 5.Test error rate (%) on character-by-character DBpedia. Test error LSTM Full Backprop (Dai & Le, 2015) 13.64 char-CNN (Zhang et al., 2015) 1.66 CNN+RNN (Xiao & Cho, 2016) 1.43 29-layer CNN (Conneau et al., 2016) 1.29 LM-LSTM (Dai & Le, 2015) 1.50 SA-LSTM(Dai & Le, 2015) 2.34 r-LSTM”