Convolutional Neural Networks for Sentence Classification
By Yoon Kim. 2014
14,347 citations recorded by the source corpus.
We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Graham Neubig
- Yoshua Bengio
- Yejin Choi
- Noah A. Smith
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Luke S. Zettlemoyer
- Yejin Choi
- Richard Socher
- Percy Liang
- Noah A. Smith
Citations in the corpus
- A Structured Self-attentive Sentence Embedding (2017, background): “, 2013) and convolutional networks (Kalchbrenner et al., 2014; dos Santos & Gatti, 2014; Kim, 2014) as an intermediate step in creating sentence representations to solve a wide variety of tasks including classification and ranking (Yin & Schütze, 2015; Palangi et al.”
- An efficient framework for learning sentence representations (2018, methodology): “Drawing inspiration from this model, Gan et al. (2016) explore the use of convolutional neural network (CNN) encoders.”
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, background): “Recently, attempts have been made to replace the recurrent networks by full convolution or full attention architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b; Shen et al., 2017a).”
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “These encoders are typically implemented as convolutional (Kim, 2014), recursive (Socher et al.”
- Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “Word Embedding Layer maps each word to a vector space using a pre-trained word embedding model.”
- Semi-supervised Sequence Learning (2015, background): “5% CNN-non-static (ConvNet with word vectors from word2vec Google News) [13] 18.”
- Skip-Thought Vectors (2015, background): “In recent years, several approaches have been developed for learning composition operators that map word vectors to sentence vectors including recursive networks [1], recurrent networks [2], convolutional networks [3, 4] and recursive-convolutional methods [5, 6] among others.”
- To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “Word representations are often fixed and fed into a task specific model ( ), although can provide improvements (Kim, 2014).”
- Generating Sentences from a Continuous Space (2015, methodology): “, 2015) and a cnn classifier (Kim, 2014) both of which, unlike our model, are optimized end-to-end.”
- Fine-tuned Language Models for Text Classification (2018, background): “Given sufficient data is available, fine-tuning representations outperforms freezing them; similar observations have been made for word embeddings (Kim, 2014).”
- Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, methodology): “To learn document features for text classification task, several methods have been proposed— Kim (2014) first learn sentence representations followed by combining them to learn document features.”
- Deep Pyramid Convolutional Neural Networks for Text Categorization (2017, methodology): “Unlike (Kim, 2014; Zhang et al., 2015; Conneau et al., 2016), variable-sized documents were handled as variable-sized without any shortening or padding; however, the vocabulary size was limited to 30K words.”
- Adversarial Training Methods for Semi-Supervised Text Classification (2016, neutral): “*(Kim, 2014) 18.5% AdaSent*(Zhao et al., 2015) 16.9% SA-LSTM † (Dai & Le, 2015) 16.7%”
- Character-level Convolutional Networks for Text Classification (2015, methodology): “It has been shown that ConvNets can be directly applied to distributed [6] [16] or discrete [13] embedding of words, without any knowledge on the syntactic or semantic structures of a language.”
- Coarse-to-Fine Question Answering for Long Documents (2016, methodology): “Following previous work (Kim, 2014; Kalchbrenner et al., 2014), we concatenate the embeddings of tokens in the query x and the”
- On Sampling Strategies for Neural Network-based Collaborative Filtering (2017, methodology): “For CNN, we adopt the structure similar in [15], and use 50 filters with filter size of 3.”
- Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “In the CNN part, convolutions with different filter widths are applied independently as in (Kim, 2014).”