Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network
By Misha Denil, Alban Demiraj, Nal Kalchbrenner, Phil Blunsom, Nando de Freitas. 2014
101 citations recorded by the source corpus.
Capturing the compositional process which maps the meaning of words to that of documents is a central challenge for researchers in Natural Language Processing and Information Retrieval. We introduce a model that is able to represent the meaning of documents by embedding them in a low dimensional vector space, while preserving distinctions of word and sentence order crucial for capturing nuanced semantics. Our model is based on an extended Dynamic Convolution Neural Network, which learns convolution filters at both the sentence and document level, hierarchically learning to capture and compose low level lexical features into high level semantic concepts. We demonstrate the effectiveness of this model on a range of document modelling tasks, achieving strong results with no feature engineering and with a more compact model. Inspired by recent advances in visualising deep convolution networks for computer vision, we present a novel visualisation technique for our document networks which not only provides insight into their learning process, but also can be interpreted to produce a compelling automatic summarisation system for texts.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Erik Cambria
- Graham Neubig
- Noah A. Smith
- Jianfeng Gao
- Luke S. Zettlemoyer
- Xipeng Qiu
In Computer Vision and Pattern Recognition
- Erik Cambria
- Soujanya Poria
- Percy Liang
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Devamanyu Hazarika
- Zhiyuan Liu
In Natural Language Processing
- Graham Neubig
- Hinrich Schütze
- Erik Cambria
- Mohit Bansal
- Xipeng Qiu
- Jianfeng Gao
- Noah A. Smith
- Luke S. Zettlemoyer
In Summarization
Citations in the corpus
- From captions to visual concepts and back (2014, neutral): “This is consistent with recent studies [41, 18, 6] in the area of natural language processing with neural networks.”
- Recent Trends in Deep Learning Based Natural Language Processing (2017, methodology): “For example, [53] applied DCNN to map meanings of words that constitute a sentence to that of documents for summarization.”
- Document Modeling with Gated Recurrent Neural Network for Sentiment Classification (2015, methodology): “01) , and set learning rate as 0.03.”
- A Convolutional Attention Network for Extreme Summarization of Source Code (2016, background): “Closely related is the work of Denil et al. (2014) that learns representations of documents using convolution but uses the network activations to summarize a document rather than an attentional model.”
- Document Context Language Models (2015, methodology): “Related convolutional architectures for document modeling are considered by Denil et al. (2014) and Tang et al. (2015).”
- A C-LSTM Neural Network for Text Classification (2015, methodology): “CNNs have been successfully combined with both sequence-based model (Denil et al., 2014; Kalchbrenner et al., 2014) and tree-structured”
- Multichannel Variable-Size Convolution for Sentence Classification (2015, background): “Hence, convolution neural networks (CNN) are getting increasing attention, for they are able to model long-range dependencies in sentences via hierarchical structures (Dos Santos and Gatti, 2014; Kim, 2014; Denil et al., 2014).”
- Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference (2018, background): “There are a few attempts toward explaining/interpreting deep learning-based models, mostly by visualizing the representation of words and/or hidden states, and their importances (via saliency or erasure) on shallow tasks like sentiment analysis and POS tagging (Bahdanau et al., 2014; Denil et al., 2”
- Multi-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents (2015, methodology): “Denil et al. (2014) used two-level version of DCNN (Kalchbrenner et al., 2014) to model documents.”
- Learning to Align the Source Code to the Compiled Object Code (2017, background): “Such tasks include sequence encoding (Zhang et al., 2015; Lee et al., 2016), sentiment prediction (Blunsom et al., 2014), document summarization (Denil et al., 2014) and translation (Gehring et al., 2017).”
- Weighted Neural Bag-of-n-grams Model: New Baselines for Text Classification (2016, methodology): “(Mesnil et al., 2014) 91.87 Paragraph Vector(Mesnil et al., 2014) 88.73 DAN(Iyyer et al., 2015) 89.40 DV-tri(Li et al., 2015a) 92.14 our model 92.95 our ensemble 93.51 sequential word2vec-LSTM(Dai and Le, 2015) 90.00 SA-LSTM(Dai and Le, 2015) 92.76 seq2-bown-CNN(Johnson and Zhang, 2015a) 92.33 CNN+u”
- Modeling Relational Information in Question-Answer Pairs with Convolutional Neural Networks (2016, methodology): “Hence, similar to (Denil et al., 2014; Kim, 2014; Yu et al., 2014) we keep the word embeddings fixed and initialize the word matrix W from an unsupervised neural language model.”
- Generating Sentences from Semantic Vector Space Representations (2014, background): “Distributed vector space models have recently shown success at capturing the semantic meanings of words [2, 15, 14], phrases and sentences [18, 16, 12], and even full documents [13, 3].”
- Understanding convolutional neural networks with a mathematical model (2016, background): “Keywords: Convolutional Neural Network (CNN), Nonlinear Activation, RECOS Model, Rectified Linear Unit (ReLU), MNIST Dataset.”
- Modelling input texts: from Tree Kernels to Deep Learning (2015, methodology): “Hence, similar to [38, 76, 189] we keep the word embeddings fixed and initialize the word matrix W from an unsupervised neural language model.”
- Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks (2015, methodology): “Hence, similar to [11, 19, 38] we keep the word embeddings fixed and initialize the word matrix WWW from an un-supervised neural language model.”