Convolutional Sequence to Sequence Learning

By Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann Dauphin. 2017

3,545 citations recorded by the source corpus.

The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, methodology): “We also experimented with using learned positional embeddings [8] instead, and found that the two versions produced nearly identical results (see Table 3 row (E)).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “Convolutional architectures have also been shown to provide competitive results for sequence modeling including sequence-to-sequence machine translation (Gehring et al., 2017).”
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, background): “Recently, attempts have been made to replace the recurrent networks by full convolution or full attention architectures (Kim, 2014; Gehring et al., 2017; Vaswani et al., 2017b; Shen et al., 2017a).”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “…2018; He et al., 2018; Hassan et al., 2018; Song et al., 2018; Shen et al., 2018), text summarization (Ayana et al., 2016; Suzuki & Nagata, 2017; Gehring et al., 2017), question answering (Yuan et al., 2017; Fedus et al., 2018) and conversational response generation (Shang et al., 2015; Vinyals…”
Generating Long Sequences with Sparse Transformers (2019, methodology): “Other works have investigated attention with multiple ”hops”, such as (Sukhbaatar et al., 2015) and (Gehring et al., 2017).”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “More recently, architectures without any recurrent structures (Gehring et al., 2017; Vaswani et al., 2017) have been proposed and shown to speed up training while achieving state-of-the-art performance.”
Universal Transformers (2018, background): “Convolutional and fully-attentional feed-forward architectures like the Transformer model have recently emerged as viable alternatives to recurrent neural networks (RNNs) for a range of sequence modeling tasks, notably machine translation [9, 31].”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Models that use only convolution (Gehring et al., 2017) or attention (Vaswani et al.”
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, methodology): “Gehring et al. (2017) developed convolutional machine translation models augmented with attention over the input sentence.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “In recent years, several revolutionary architectures have been developed to improve MT quality (Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018a; Wu et al., 2019).”
Self-Attention with Relative Position Representations (2018, background): “They have been shown to still beneﬁt from position encodings (Gehring et al., 2017), however.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “, 2015), convolutional models (Gehring et al., 2017; Wu et al., 2019) and Transformer (Vaswani et al.”
Scaling Neural Machine Translation (2018, methodology): “For En–Fr, we train on WMT’14 and borrow the setup of Gehring et al. (2017) with 36M training sentence pairs.”
RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, methodology): “[2017] was added to the contextual representations, while a trainable absolute position encoding Gehring et al. [2017], Devlin et al. [2019], Lan et al.”
Big Bird: Transformers for Longer Sequences (2020, neutral): “[29] J.”
MLSUM: The Multilingual Summarization Corpus (2020, neutral): “…this observation is in line with the results reported for Machine Translation: the Transformer (Vaswani et al., 2017) outperforms signiﬁcantly ConvS2S (Gehring et al., 2017) for English to German but obtains comparable results for English to French – see Table 2 in Vaswani et al. (2017).”
Tagged Back-Translation (2019, background): “Neural Machine Translation (NMT) has made considerable progress in recent years (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017).”
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, methodology): “Recently, a convolutional alternative to sequence modeling has been proposed showing promise for machine translation (Gehring et al., 2017a,b) and story generation (Fan et al., 2018).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “, 2012; Collobert, Puhrsch, and Synnaeve, 2016), computer vision (Krizhevsky, Sutskever, and Hinton, 2012), and machine translation (Sutskever, Vinyals, and Le, 2014; Gehring et al., 2017).”
ELI5: Long Form Question Answering (2019, methodology): “We use the Transformer implementation of fairseq-py (Gehring et al., 2017) and train with the big architecture following the details in (Vaswani et al.”