A Convolutional Encoder Model for Neural Machine Translation

By Jonas Gehring, Michael Auli, David Grangier, Yann Dauphin. 2016

466 citations recorded by the source corpus.

The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence. We present a faster and simpler architecture based on a succession of convolutional layers. This allows to encode the source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies. On WMT’16 English-Romanian translation we achieve competitive accuracy to the state-of-the-art and on WMT’15 English-German we outperform several recently published results. Our models obtain almost the same accuracy as a very deep LSTM setup on WMT’14 English-French translation. We speed up CPU decoding by more than two times at the same or higher accuracy as a strong bi-directional LSTM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, background): “Architec-1 tures which are partially convolutional have shown strong performance on larger tasks but their decoder is still recurrent (Gehring et al., 2016).”
Massive Exploration of Neural Machine Translation Architectures (2017, neutral): “Our model is outper-formed only by (Wu et al., 2016 (Jean et al., 2015), RNNSearch-LV (Jean et al., 2015), BPE (Sennrich et al., 2016b), BPE-Char (Chung et al., 2016), Deep-Att (Zhou et al., 2016), Luong (Lu-ong et al., 2015a), Deep-Conv (Gehring et al., 2016), GNMT (Wu et al., 2016), and OpenNMT (K”
Graph Attention Networks (2017, methodology): “Convolutional Neural Networks (CNNs) have been successfully applied to tackle problems such as image classiﬁcation (He et al., 2016), semantic segmentation (J´egou et al., 2017) or machine translation (Gehring et al., 2016), where the underlying data representation has a grid-like structure.”
A Call for Clarity in Reporting BLEU Scores (2018, methodology): “ic lc Bahdanau et al. (2014) (unclear) Luong et al. (2015b) user or metric (unclear) Jean et al. (2015) user Wu et al. (2016) user or user lc (unclear) Vaswani et al. (2017) user or user lc (unclear) Gehring et al. (2017) user, metric Table 2: Benchmarks set by well-cited papers use different BLEU c”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “Recently, a convolutional alternative to sequence modeling has been proposed showing promise for machine translation (Gehring et al., 2017a,b) and story generation (Fan et al., 2018).”
Relation Networks for Object Detection (2017, methodology): “Attention modules in NLP and physical system modeling Attention modules have recently been successfully applied in the NLP field [20, 21, 6, 56] and in physical system modeling [2, 57, 31, 45, 13, 42].”
Compositionality Decomposed: How do Neural Networks Generalise? (2019, methodology): “We compare three currently popular neural architectures for sequence-to-sequence language processing tasks: a recurrent architecture (LSTMS2S) [Sutskever et al. , 2014], a convolution-based architecture (ConvS2S) [Gehring et al. , 2017] and a transformer model (Transformer) [Vaswani et al. , 2017].”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “…in sequence modeling through recurrent neural networks (RNN; Sutskever et al. 2014; Bahdanau et al. 2015; Wu et al. 2016), convolutional networks (CNN; Kalchbrenner et al. 2016; Gehring et al. 2016; 2017; Kaiser et al. 2017) and self-attention models (Paulus et al., 2017; Vaswani et al., 2017).”
Fully Convolutional Speech Recognition (2018, background): “Recent work on convolutional neural network architectures showed that they are competitive with recurrent architectures even on tasks where modeling long-range dependencies is critical, such as language modeling [1], machine translation [2, 3] and speech synthesis [4].”
Sharp Minima Can Generalize For Deep Nets (2017, background): “…Szegedy et al., 2015; He et al., 2016), machine translation (e.g. Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016; Gehring et al., 2016) and speech recognition (e.g. Graves et al., 2013; Hannun et al., 2014; Chorowski et al., 2015; Chan et al., 2016; Collobert et…”
Improving Neural Language Modeling via Adversarial Training (2019, methodology): “Speciﬁcally, we use an RNN encoder-decoder framework (Cho et al., 2014; Gehring et al., 2017b; Vaswani et al., 2017), upon which we apply our adversarial MLE training that learns to translate.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “Particularly inspiring for our work are the recent applications of convolutional architectures to machine translation (Kalchbrenner et al., 2016; Gehring et al., 2017a;b), audio synthesis (van den Oord et al., 2016), and language modeling (Dauphin et al., 2017).”
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation (2017, methodology): “Instead of the approach used by Gehring et al. (2016) (i.e. stacking multiple CNN layers on top of each other), we use a GCN to enrich the one-layer CNN representations.”
Deformable ConvNets V2: More Deformable, Better Results (2018, background): “Relation Networks and Attention Modules are first proposed in natural language processing [13, 14, 3, 35] and physical system modeling [2, 37, 22, 34, 9, 31].”
Weighted Transformer Network for Machine Translation (2017, methodology): “Several architectures have been proposed to reduce the computational load associated with recurrence-based computation (Gehring et al., 2016; 2017; Kaiser & Bengio, 2016; Kalchbrenner et al., 2016).”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, methodology): “…et al., 2014; Cho et al., 2014), to an improved version featuring an attention mechanism (Bahdanau et al., 2015; Luong et al., 2015a), to models that only use attention instead of recurrent layers (Vaswani et al., 2017) and models that apply convolution networks (Gehring et al., 2017a,b).”
Classical Structured Prediction Losses for Sequence to Sequence Learning (2017, methodology): “In our setting, ofﬂine generation has 26 times higher throughput than the online generation setting, despite the high inference speed of fairseq (Gehring et al., 2017b).”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, background): “Indeed, modern neural MT systems can achieve near human-level translation performance on language pairs for which sufﬁcient parallel training resources exist (e.g., Chinese–English translation (Hassan et al., 2018) and English–French translation (Gehring et al., 2016; Ott et al., 2018a).”