Comparative Study of CNN and RNN for Natural Language Processing

By Wenpeng Yin, Katharina Kann, Mo Yu, Hinrich Schütze. 2017

1,101 citations recorded by the source corpus.

Deep neural networks (DNN) have revolutionized the field of natural language processing (NLP). Convolutional neural network (CNN) and recurrent neural network (RNN), the two main types of DNN architectures, are widely explored to handle various NLP tasks. CNN is supposed to be good at extracting position-invariant features and RNN at modeling units in sequence. The state of the art on many NLP tasks often switches due to the battle between CNNs and RNNs. This work is the first systematic comparison of CNN and RNN on a wide range of representative NLP tasks, aiming to give basic guidance for DNN selection.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

VEGA: Towards an End-to-End Configurable AutoML Pipeline (2020, methodology): “…method for artiﬁcial intelligence in various tasks, such as computer vision (He et al., 2016; Ren et al., 2015), natural language processing (NLP) (Yin et al., 2017; Devlin et al., 2018), autonomous driving (Al-Qizwini et al., 2017; Kuutti et al., 2020) and recommender systems (Guo et al., 2017;…”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “(Yin et al. (2017) have reported a comparison of convolutional and recurrent networks for sentence-level and document-level classiﬁcation tasks.”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “[83] provided interesting insights on the comparative performance between RNNs and CNNs.”
Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures (2018, background): “2 Related work Yin et al. (2017) are the ﬁrst to compare CNNs, LSTMs and GRUs on several NLP tasks.”
Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism (2020, background): “Figure 2 shows the basic architecture of the 1D CNN [14].”
Improving Natural Language Inference Using External Knowledge in the Science Questions Domain (2018, result): “Similar to the results shown in (Yin et al. 2017), using GRUs with match-LSTM slightly improved the accuracy (particularly on the dev set).”
An Introductory Review of Deep Learning for Prediction Models With Big Data (2020, background): “Hence, CNNs show very strong potential in performing classiﬁcation tasks where successful predictions heavily rely on extracting key information from input text (Yin et al., 2017).”
Joint Modelling of Emotion and Abusive Language Detection (2020, background): “This can be attributed to the fact that CNNs are well suited to extract local and position-invariant features (Yin et al., 2017).”
Augmenting Neural Machine Translation with Knowledge Graphs (2019, background): “Previous work has compared NN architectures on a variety of NLP tasks (Yin et al., 2017; Linzen et al., 2016; Bernardy and Lappin, 2017).”
Transformer Hawkes Process (2020, background): “Recently, convolutional neural network variants that are tailored for analyzing sequential data (Oord et al., 2016; Gehring et al., 2017; Yin et al., 2017) have been proposed to better capture long-term e ﬀ ects.”
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice (2020, background): “Common types of DL architectures include deep neural networks (DNNs), feedforward neural networks (FFNNs), deep belief networks (DBNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs) and many more [63].”
Deep Learning for Aspect-Based Sentiment Analysis: A Comparative Review (2019, background): “Yin et al. (2017) evaluated sentiment classification, question answering and POS tagging and concluded that model choice depended on the global semantics of a classification task and that, therefore, the focus should be on hyper-parameters such as layer size.”
Deep Learning for Time Series Forecasting: The Electric Load Case (2019, background): “Several works in the natural language processing community show that GRUs perform comparably to LSTM but train generally faster due to the lighter computation [52, 53].”
Deep reinforcement learning for time series: playing idealized trading games (2018, methodology): “As both RNN and CNN have been used in sequence modeling as introduced above, Yin et al. [21] compared RNN and CNN regarding their performance for natural language processing.”
Unknown Intent Detection Using Gaussian Mixture Model with an Application to Zero-shot Intent Classification (2020, background): “Both types of methods have shown promising results in practice (Yin et al., 2017).”