genCNN: A Convolutional Architecture for Word Sequence Prediction

By Mingxuan Wang, Zhengdong Lu, Hang Li, Wenbin Jiang, Qun Liu. 2015

29 citations recorded by the source corpus.

We propose a novel convolutional architecture, named $gen$CNN, for word sequence prediction. Different from previous work on neural network-based language modeling and generation (e.g., RNN or LSTM), we choose not to greedily summarize the history of words as a fixed length vector. Instead, we use a convolutional neural network to predict the next word with the history of words of variable length. Also different from the existing feedforward networks for language modeling, our model can effectively fuse the local correlation and global correlation in the word sequence, with a convolution-gating strategy specifically designed for the task. We argue that our model can give adequate representation of the history, and therefore can naturally exploit both the short and long range dependencies. Our model is fast, easy to train, and readily parallelized. Our extensive experiments on text generation and $n$-best re-ranking in machine translation show that $gen$CNN outperforms the state-of-the-arts with big margins.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alexander M. Rush
2. G. Wang
3. Jianfei Cai
4. Mohit Bansal
5. Yoon Kim
6. Kyunghyun Cho
7. Mirella Lapata
8. Hui Jiang

Citations in the corpus

• Recent advances in convolutional neural networks (2015, methodology): “genCNN [306] is a convolutional architecture for sequence prediction, which uses separate gating networks to replace the max-pooling operations.”
• Character-Aware Neural Language Models (2015, methodology): “Some example architectures include feed-forward (Bengio, Ducharme, and Vincent 2003), recurrent (Mikolov et al. 2010), sum-product (Cheng et al. 2014), log-bilinear (Mnih and Hinton 2007), and convolutional (Wang et al. 2015) networks.”
• Recurrent Highway Networks with Language CNN for Image Captioning (2016, methodology): “Such a method is inspired by papers in natural language processing which have shown CNN is very powerful for text representation [15, 14, 10, 12, 36, 8].”
• Higher Order Recurrent Neural Networks (2016, methodology): “Besides, short-cut skipping connections were considered earlier in Wermter (1992), and more recently have been found useful in learning very deep feed-forward neural networks as well, such as Lee et al.”