Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers

By Yijun Xiao, Kyunghyun Cho. 2016

223 citations recorded by the source corpus.

Document classification tasks were primarily tackled at word level. Recent research that works with character-level inputs shows several benefits over word-level approaches such as natural incorporation of morphemes and better handling of rare words. We propose a neural network architecture that utilizes both convolution and recurrent layers to efficiently encode character inputs. We validate the proposed model on eight large scale document classification tasks and compare with character-level convolution-only models. It achieves comparable performances with much less parameters.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

FastText.zip: Compressing text classification models (2016, methodology): “They are attractive models for text classiﬁcation because they achieve similar performance with less memory usage than linear models (Xiao & Cho, 2016).”
Bag of Tricks for Efficient Text Classification (2016, background): “…and TFIDF baselines from Zhang et al. (2015), as well as the character level convolutional model (char-CNN) of Zhang and LeCun (2015), the character based convolution recurrent network (char-CRNN) of (Xiao and Cho, 2016) and the very deep convolutional network (VDCNN) of Conneau et al. (2016).”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, methodology): “3 Towards Character-Level Translation”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, methodology): “to text, most notably in text classification (Zhang et al., 2015; Xiao and Cho, 2016).”
Very Deep Convolutional Networks for Text Classification (2016, methodology): “The same idea was recently applied to sentence classiﬁcation (Xiao and Cho, 2016).”
Quasi-Recurrent Neural Networks (2016, background): “Xiao & Cho (2016) also tackle text classiﬁcation by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM.”
Very Deep Convolutional Networks for Natural Language Processing (2016, methodology): “The initial motivation of the authors was to obtain the same performance as Zhang et al. [20] with networks which have signiﬁcantly fewer parameters.”
Neural Network Methods for Natural Language Processing (2017, methodology): “Such an approach is explored by Xiao and Cho [2016] on several document classification tasks.”
Byte-Level Machine Reading Across Morphologically Varied Languages (2018, methodology): “A graphical illustration of this model is provided in Figure 2c. Convolutional-recurrent The convolutional-recurrent model is based on the model in (Xiao and Cho 2016).”
Towards a Seamless Integration of Word Senses into Downstream NLP Applications (2017, methodology): “The general architecture of the model was the same for both tasks, with slight variations in hyperparameters given the different natures of the tasks, following the values suggested by Kim (2014) and Xiao and Cho (2016) for the two tasks.”
Depression and Self-Harm Risk Assessment in Online Forums (2017, methodology): “While our model shares similarities with CNN-based models in prior work (Kalchbrenner et al., 2014; Kim, 2014; Xiao and Cho, 2016), it focuses on learning representations of user’s posts and combining the post representations into an overall representation of the user’s activity.”
Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce (2016, methodology): “In particular, Convolutional neural networks (CNNs) (Kim, 2014; Zhang et al., 2015; Conneau et al., 2016) and Recurrent NNs (Lai et al., 2015; Pyo et al., 2010; Xiao & Cho, 2016) can efﬁciently capture the sequentiality of the text.”
Densely Connected CNN with Multi-scale Feature Attention for Text Classification (2018, unknown): “4 - RNN-CNN CNN with Recurrent Layer∗ [Xiao and Cho, 2016] - 91.”
Efficient Large-Scale Domain Classification with Personalized Attention (2018, methodology): “Character-informed sequence models have also been explored for simple text classiﬁcation with small sets of classes by Xiao and Cho (2016).”
Generative and Discriminative Text Classification with Recurrent Neural Networks (2017, methodology): “Our discriminative LSTM model is competitive with other discriminative models based on logistic regression (Zhang et al., 2015; Joulin et al., 2016) or convolutional neural networks (Zhang et al., 2015; Xiao & Cho, 2016; Conneau et al., 2016).”
Semi-Supervised Deep Learning Using Pseudo Labels for Hyperspectral Image Classification (2018, methodology): “The CRNN is a hybrid model of convolutional and recurrent neural networks [22], [50], [51].”