Convolutional Neural Network Language Models

By Ngoc-Quan Pham, Germán Kruszewski, Gemma Boleda. 2016

40 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) have shown to yield very strong results in several Computer Vision tasks. Their application to language has received much less attention, and it has mainly focused on static classiﬁcation tasks, such as sentence classiﬁcation for Sentiment Analysis or relation extraction. In this work, we study the application of CNNs to language modeling, a dynamic, sequential prediction task that needs models to capture local as well as long-range dependency information. Our contribution is twofold. First, we show that CNNs achieve 11-26% better absolute performance than feed-forward neural language models, demonstrating their potential for language representation even in sequential tasks. As for recurrent models, our model outperforms RNNs but is below state of the art LSTM models. Second, we gain some understanding of the behavior of the model, showing that CNNs in language act as feature detectors at a high level of abstraction, like in Computer Vision, and that the model can prof-itably use information from as far as 16 words before the target.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Convolutional Encoder Model for Neural Machine Translation (2016, background): “We experiment on several standard datasets and compare our approach to variants of recurrent encoders such as uni-directional and bi-directional LSTMs.”
Information-Theoretic Probing with Minimum Description Length (2020, background): “…2018; Raganato and Tiede-mann, 2018; Voita et al., 2019b; Clark et al., 2019; Kovaleva et al., 2019) or individual neurons (Karpa-thy et al., 2015; Pham et al., 2016; Bau et al., 2019), (ii) looking at model predictions using manually deﬁned templates, either evaluating sensitivity to speciﬁc…”
FLIPDIAL: A Generative Model for Two-Way Visual Dialogue (2018, background): “Recently however, there has been considerable interest in convolutional models of language [3, 11, 14, 21], which have shown to perform at least as well as recurrent models, if not better, on a number of different tasks.”
Language modeling with neural trans-dimensional random fields (2017, methodology): “It is shown in [18] that applying convolutional layers in FNNs performs better than conventional FNN LMs but is below LSTM LMs. Convolutional layers can also be used within RNNs, e.g. as studied in [19], 1-”
Residual Conv-Deconv Grid Network for Semantic Segmentation (2017, background): “07958v2 Convolutional Neural Networks (CNN) have become tremendously popular for a huge number of applications [1, 16, 20] since the success of AlexNet [9] in 2012.”
Artificial Intelligence for the Metaverse: A Survey (2022, background): “short paragraphs, especially being efficient to specific and complicated word patterns [23].”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, methodology): “A CNN used recently in language modeling replaced the pooling layers with fully connected layers [58].”
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models (2019, methodology): “For the language data, we employ a CNN encoder and decoder following Kalchbrenner et al. (2014); Massiceti et al. (2018b); Pham et al. (2016), learning an embedding for words in the process.”
End-to-End Speech Recognition: A Survey (2023, methodology): “4) ConvLM: Convolutional neural networks (ConvLM) have also been applied to LMs [291], [292], [293].”