Recurrent Convolutional Neural Networks for Discourse Compositionality

By Nal Kalchbrenner, Phil Blunsom. 2013

249 citations recorded by the source corpus.

The compositionality of meaning extends beyond the single sentence. Just as words combine to form the meaning of sentences, so do sentences combine to form the meaning of paragraphs, dialogues and general discourse. We introduce both a sentence model and a discourse model corresponding to the two levels of compositionality. The sentence model adopts convolution as the central operation for composing semantic vectors and is based on a novel hierarchical convolutional neural network. The discourse model extends the sentence model and is based on a recurrent neural network that is conditioned in a novel way both on the current sentence and on the current speaker. The discourse model is able to capture both the sequentiality of sentences and the interaction between different speakers. Without feature engineering or pretraining and with simple greedy decoding, the discourse model coupled to the sentence model obtains state of the art performance on a dialogue act classification experiment.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Convolutional Neural Network for Modelling Sentences (2014, methodology): “Finally, a further class of neural sentence models is based on the convolution operation and the TDNN architecture (Collobert and Weston, 2008; Kalchbrenner and Blunsom, 2013b).”
Recurrent Continuous Translation Models (2013, methodology): “For a formal treatment of the construction, see (Kalchbrenner and Blunsom, 2013).”
Deep Unordered Composition Rivals Syntactic Methods for Text Classification (2015, neutral): “Previously published experimental results have shown that syntactic functions outperform unordered functions on many tasks (Socher et al., 2013b; Kalchbrenner and Blunsom, 2013).”
WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia (2016, background): “The highest score of 71.8% is achieved by a sequence to sequence model (Kalchbrenner and Blunsom, 2013; Cho et al., 2014) operating on word-level input and output sequences, with special handing for out-of-vocabulary words.”
Neural Summarization by Extracting Sentences and Words (2016, background): “lutional neural network (CNN) with a max-overtime pooling operation (Kalchbrenner and Blunsom, 2013; Zhang and Lapata, 2014; Kim et al., 2016).”
Recurrent Convolutional Neural Networks for Text Classification (2015, background): “However, previous studies on CNNs tends to use simple convolutional kernels such as a ﬁxed window (Collobert et al. 2011; Kalchbrenner and Blunsom 2013).”
Multimodal Convolutional Neural Networks for Matching Image and Sentence (2015, methodology): “8]. For sentence, there is a thread of neural networks for the sentence representation, such as CNN [17,20], time-delay neural network [4], recursive neural network [15], and recurrent neural network [16,37,29,40]. The obtained sentence representation can be used for the sentence classiﬁcation [20],”
Multilingual Distributed Representations without Word Alignment (2013, methodology): “Such models include recursive autoencoders [24], matrix-vector recursive neural networks [25], untied recursive neural networks [14] or convolutional networks [15].”
Multilingual Models for Compositional Distributed Semantics (2014, background): “, 2003), paraphrase detection (Erk and Padó, 2008), and dialogue analysis (Kalchbrenner and Blunsom, 2013).”
Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model (2015, methodology): “Supervision Unlike many recent composition models (Kalchbrenner and Blunsom, 2013; Kalch-brenner et al., 2014; Socher et al., 2012; Socher et al., 2013, among others), the context-prediction objective of C-PHRASE does not require annotated data, and it is meant to provide general-purpose…”
A Model of Coherence Based on Distributed Sentence Representation (2014, background): “More recent researches have begun looking at higher level distributed representations that transcend the token level, such as sentence-level (Le and Mikolov, 2014) or even discourse-level (Kalchbrenner and Blunsom, 2013) aspects.”
Abstractive Dialogue Summarization with Sentence-Gated Modeling Optimized by Dialogue Acts (2018, methodology): “A dialogue instance in the dataset built from the AMI meeting corpus. rent neural network (RNN) [33, 34, 35, 36].”
Compositional Operators in Distributional Semantics (2014, methodology): “Kalchbrenner and Blunsom (2013b), for example, model sentential compositionality using a convolutional neural network in an elementwise fashion.”
A Deep Architecture for Semantic Parsing (2014, methodology): “Recent work in dialogue act labelling (Kalchbrenner and Blunsom, 2013b) and in machine translation (Kalchbrenner and Blun-som, 2013a) has demonstrated that a particular kind of neural language model based on recurrent neural networks (Mikolov et al., 2010; Sutskever et al., 2011) could be extended…”
Extraction of Salient Sentences from Labelled Documents (2014, background): “…et al., 2014; Sutskever et al., 2014; Hermann & Blunsom, 2014), question answering (Bordes et al., 2014; Weston et al., 2014), dialogue systems (Kalchbrenner & Blunsom, 2013b), sentiment analysis (Socher et al., 2011; 2012; Hermann & Blunsom, 2013), and other natural language processing tasks…”
Evaluating Neural Word Representations in Tensor-Based Compositional Settings (2014, methodology): “…in compositional models involving general mathematical composition operators, rather than in the more task- or domain-specific deep-learning compositional settings they have generally been used with so far (for example, by Socher et al. (2012), Kalchbrenner and Blunsom (2013) and many others).”
Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks (2015, background): “Recent improvements of convolutional neural networks (CNNs) have been proven to be efﬁcient for capturing syntactic and semantics between words within a sentence (Col-lobert et al., 2011; Kalchbrenner and Blunsom, 2013; Zeng et al., 2014) for NLP tasks.”
If Beam Search Is the Answer, What Was the Question? (2020, background): “As such, it became the nat-ural choice for decoding neural probabilistic text generators—whose design makes evaluating the full search space impossible (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Vinyals and Le, 2015; Yin et al., 2016).”
Have You Lost the Thread? Discovering Ongoing Conversations in Scattered Dialog Blocks (2017, background): “Many models have been proposed (Mitchell and Lapata 2008; Baroni and Zamparelli 2010; Clark et al. 2008; Grefenstette and Sadrzadeh 2011; Zanzotto et al. 2010; Socher et al. 2011, 2012; Kalchbrenner and Blunsom 2013).”