A Convolutional Neural Network for Modelling Sentences

By Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom. 2014

3,651 citations recorded by the source corpus.

The ability to accurately represent sentences is central to language understanding. We describe a convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pooling, a global pooling operation over linear sequences. The network handles input sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations. The network does not rely on a parse tree and is easily applicable to any language. We test the DCNN in four experiments: small scale binary and multi-class sentiment prediction, six-way question classification and Twitter sentiment prediction by distant supervision. The network achieves excellent performance in the first three tasks and a greater than 25% error reduction in the last task with respect to the strongest baseline.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “, 2013) and convolutional networks (Kalchbrenner et al., 2014; dos Santos & Gatti, 2014; Kim, 2014) as an intermediate step in creating sentence representations to solve a wide variety of tasks including classification and ranking (Yin & Schütze, 2015; Palangi et al.”
Exploring the Limits of Language Modeling (2016, neutral): “Lastly, recently proposed sequence-to-sequence models employ conditional language models (Mikolov & Zweig, 2012) as their key component to solve diverse tasks like machine translation (Sutskever et al., 2014; Cho et al., 2014; Kalchbrenner et al., 2014) or video generation (Srivastava et al.”
Long Short-Term Memory-Networks for Machine Reading (2016, neutral): “6 DCNN (Blunsom et al., 2014) 48.”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “This excludes innovative supervised sentencelevel architectures including (Socher et al., 2011; Kalchbrenner et al., 2014) and many others.”
Skip-Thought Vectors (2015, background): “In recent years, several approaches have been developed for learning composition operators that map word vectors to sentence vectors including recursive networks [1], recurrent networks [2], convolutional networks [3, 4] and recursive-convolutional methods [5, 6] among others.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “The original Transformer consisted of an encoder-decoder architecture and was intended for sequence-to-sequence [Sutskever et al., 2014; Kalchbrenner et al., 2014] tasks.”
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “While we adopt a bag-of-words approach for practical reasons (memory and run-time), our multi-task framework is extensible to other methods for sentence/document representations, such as those based on convolutional networks (Kalchbrenner et al., 2014; Shen et al., 2014; Gao et al., 2014b), parse tr”
Teaching Machines to Read and Comprehend (2015, methodology): “This includes classiﬁcation tasks such as sentiment analysis [15] or POS tagging [16], as well as generative problems such as language modelling or machine translation [17].”
A Deep Relevance Matching Model for Ad-hoc Retrieval (2016, methodology): “For example, MV-LSTM [26] used K-max pooling strategy [13] to select the top K strongest interaction signals from the matching matrix as the input of a MLP.”
Coarse-to-Fine Question Answering for Long Documents (2016, methodology): “Following previous work (Kim, 2014; Kalchbrenner et al., 2014), we concatenate the embeddings of tokens in the query x and the”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, methodology): “A text-to-text model follows the sequence-to-sequence framework (Sutskever et al., 2014; Kalchbrenner et al., 2014) – it is fed a sequence of discrete tokens as”
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (2017, neutral): “…well on natural language processing tasks such as machine translation (e.g. Sutskever et al., 2014; Bahdanau et al., 2015), sentiment analysis (e.g. Kalchbrenner et al., 2014; Socher et al., 2013; Le and Zuidema, 2015), semantic role labelling (e.g. Roth and Lapata, 2016) and recognizing textual…”
Convolutional Neural Networks for Sentence Classification (2014, methodology): “Even a simple model with static vectors (CNN-static) performs remarkably well, giving competitive results against the more sophisticated deep learning models that utilize complex pooling schemes (Kalchbrenner et al., 2014) or require parse trees to be computed beforehand (Socher et al., 2013).”
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015, neutral): “DCNN: Kalchbrenner et al. (2014). PVec: Le & Mikolov.”
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015, neutral): “Fine-grained Binary RAE (Socher et al., 2013) 43.2 82.4 MV-RNN (Socher et al., 2013) 44.4 82.9 RNTN (Socher et al., 2013) 45.7 85.4 DCNN (Blunsom et al., 2014) 48.5 86.8 Paragraph-Vec (Le and Mikolov, 2014) 48.7 87.8 CNN-non-static (Kim, 2014) 48.0 87.2 CNN-multichannel (Kim, 2014) 47.4 88.1 DRNN…”
Rationalizing Neural Predictions (2016, methodology): “For instance, we could use a convolutional network (Kim, 2014; Kalchbrenner et al., 2014), deep averaging network (Iyyer et al.”