Encoding Source Language with Convolutional Neural Network for Machine Translation

By Fandong Meng, Zhengdong Lu, Mingxuan Wang, Hang Li, Wenbin Jiang, Qun Liu. 2015

109 citations recorded by the source corpus.

The recently proposed neural network joint model (NNJM) (Devlin et al., 2014) augments the n-gram target language model with a heuristically chosen source context window, achieving state-of-the-art performance in SMT. In this paper, we give a more systematic treatment by summarizing the relevant source information through a convolutional architecture guided by the target information. With different guiding signals during decoding, our specifically designed convolution+gating architectures can pinpoint the parts of a source sentence that are relevant to predicting a target word, and fuse them with the context of entire source sentence to form a unified representation. This representation, together with target language words, are fed to a deep neural network (DNN) to form a stronger NNJM. Experiments on two NIST Chinese-English translation tasks show that the proposed model can achieve significant improvements over the previous NNJM by up to +1.08 BLEU points on average

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Model Architectures

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, methodology): “Gated convolutions have been previously explored for machine translation by Meng et al. (2015) but their evaluation was restricted to a small dataset and the model was used in tandem with a traditional count-based model.”
A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “Meng et al. (2015) and (Tu et al., 2015) applied convolutional models to score phrase-pairs of traditional phrasebased and dependency-based translation models.”
Tensor2Tensor for Neural Machine Translation (2018, methodology): “Convolutional architectures have been used to obtain good results in word-level neural machine translation starting from Kalchbrenner and Blunsom (2013) and later in Meng et al. (2015).”
Depthwise Separable Convolutions for Neural Machine Translation (2017, methodology): “Convolutional architectures have been used to obtain good results in word-level neural machine translation starting from [10] and later in [14].”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, methodology): “Convolutional architectures have been used to obtain good results in word-level neural machine translation starting from (Kalchbrenner & Blunsom, 2013) and later in (Meng et al., 2015).”
Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, background): “The combination of CNN and LSTM has been considered in image captioning (Karpathy and Fei-Fei, 2015), and in some recent work on machine translation (Kalchbrenner and Blunsom, 2013; Meng et al., 2015; Gehring et al., 2016).”
Deconvolutional Paragraph Representation Learning (2017, background): “These representations are typically a required first step toward more applied tasks, such as sentiment analysis [1, 2, 3, 4], machine translation [5, 6, 7], dialogue systems [8, 9, 10] and text summarization [11, 12, 13].”
Relation Classification via Multi-Level Attention CNNs (2016, methodology): “Attention mechanisms have successfully been applied to sequence-to-sequence learning tasks such as machine translation (Bahdanau et al., 2015; Meng et al., 2015) and abstractive sentence summarization (Rush et al., 2015), as well as to tasks such as modeling sentence pairs (Yin et al., 2015) and…”
Task-Oriented Multi-User Semantic Communications (2021, methodology): “The representative models include CNN based models [34], Transformer based models [24], and RNN based models, i.e., LSTM networks [35] and Gated Recurrent Units (GRU) networks [36].”
Mutual Information and Diverse Decoding Improve Neural Machine Translation (2016, methodology): “Similar convolutional networks are used in (Meng et al., 2015) for encoding.”
TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays (2018, background): “Wang et al. [35] provide a recent and prominent exception, where they introduced a large scale chest X-ray dataset by processing images and their paired radiological reports (extracted from their institutional PACS database) with natural language processing (NLP) techniques.”
Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction (2018, unknown): “(2017b) outperformed deep LSTMs for machine translation 1D CNNs with gated linear units (Meng et al., 2015; Oord et al., 2016c; Dauphin et al., 2017) in both the encoder and decoder modules.”
Table-to-Text: Describing Table Region With Natural Language (2018, methodology): “The encoder can be implemented with various neural architectures such as convolutional neural network (CNN) (Meng et al. 2015; Gehring et al. 2016) and recurrent neural network (RNN) (Cho et al. 2014; Sutskever, Vinyals, and Le 2014).”
Question Answering and Question Generation as Dual Tasks (2017, methodology): “The encoder could be implemented with different neural network architectures such as convolutional neural network (Kalchbrenner and Blunsom 2013; Meng et al. 2015) and recurrent neural network (RNN) (Bahdanau, Cho, and Bengio 2014; Sutskever, Vinyals, and Le 2014).”
Unsupervised Learning of Sentence Representations using Convolutional Neural Networks (2016, methodology): “For the hierarchical CNN-LSTM model, we use mini-batches of size 8, and each paragraph is composed of 8 sentences.”