Weighted Transformer Network for Machine Translation

By Karim Ahmed, Nitish Shirish Keskar, Richard Socher. 2017

142 citations recorded by the source corpus.

State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art results on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15-40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Universal Transformers (2018, background): “The UT with a fully-connected recurrent transition function (instead of separable convolution) and without ACT improves by 0.9 BLEU over a Transformer and 0.5 BLEU over a Weighted Transformer with approximately the same number of parameters (Ahmed et al., 2017).”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, neutral): “…de,fr } + numrefs.1+smooth.exp+test.wmt14/full+tok.13a+version.1.2.9 En–De En–Fr a. Gehring et al. (2017) 25.2 40.5 b. Vaswani et al. (2017) 28.4 41.0 c. Ahmed et al. (2017) 28.9 41.4 d. Shaw et al. (2018) 29 (Grave et al., 2017), and adaptive inputs (Baevski and Auli, 2019).”
Scaling Neural Machine Translation (2018, neutral): “En–Fr a. Gehring et al. (2017) 25.2 40.5 b. Vaswani et al. (2017) 28.4 41.0 c. Ahmed et al. (2017) 28.9 41.4 d. Shaw et al. (2018) 29 nodes vs. overlap comm+bwd ).”
Are Sixteen Heads Really Better than One? (2019, background): “Since then, several extensions to the general methodology have been proposed (Ahmed et al., 2017; Shen et al., 2018).”
Efficient Transformers: A Survey (2020, background): “As such, it is only natural that a wealth of research has been dedicated to making fundamental improvements to the model over the past few years (Dehghani et al., 2018; So et al., 2019; Ahmed et al., 2017).”
The Evolved Transformer (2019, methodology): “For translation speciﬁcally, work on scaling up batch size (Ott et al., 2018; Wu et al., 2019), using relative position representations (Shaw et al., 2018), and weighting multi-head attention (Ahmed et al., 2017) have all pushed the state-of-the-art for WMT 2014 En-De and En-Fr.”
Alibaba’s Neural Machine Translation Systems for WMT18 (2018, methodology): “The heads in the multi-head attention are independent of each other, Ahmed et al. (2017) improved this with a new mechanism, namely multi-branch attention.”
The University of Cambridge’s Machine Translation Systems for WMT18 (2018, background): “…Bahdanau et al., 2015; Luong et al., 2015; Wu et al., 2016; Chen et al., 2018), convolutional models (Kalchbrenner and Blunsom, 2013; Kaiser et al., 2017; Gehring et al., 2017), and, most recently, purely (self-)attention-based models (Vaswani et al., 2017; Ahmed et al., 2017; Shaw et al., 2018).”
Data Diversification: A Simple Strategy For Neural Machine Translation (2019, background): “For instance, Shaw et al. [19] and Ahmed et al. [1] propose minor modiﬁcations to improve the original Transformer [24] with slight performance gains.”
Neural Machine Translation: A Review (2019, background): “• Various extensions to the Transformer architecture (Shaw et al., 2018; Ahmed et al., 2017; Guo et al., 2019; Medina & Kalita, 2018)”
Fully Quantized Transformer for Machine Translation (2019, background): “Nonetheless, the approach became the state-of-the-art in the ﬁeld (Ahmed et al., 2017; Ott et al., 2018; Edunov et al., 2018).”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, background): “Convolutional Seq-to-Seq [136] 25.2 Attention Is All You Need [137] 28.4 Weighted Transformer [138] 28.9 Self Attention [139] 29.2 DeepL Translation Machine 10 33.3 Back-translation [140] 35.0 can be inferred from available documents.”
Alternating Language Modeling for Cross-Lingual Pre-Training (2020, background): “• Weighted Transformer (Ahmed, Keskar, and Socher 2017): It uses self-attention branches in place of multihead attention.”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, result): “In a direction diverging from previous work, Vaswani et al. [42], [251] proposed discarding the large number of recurrent and convolutional layers and instead focusing exclusively on attention mechanisms to encode a language globally from input to output.”
Lite Transformer with Long-Short Range Attention (2020, background): “Recently, there have been a lot of variants to the transformer (Ahmed et al., 2017; Ott et al., 2018; Chen et al., 2018; Paulus et al., 2018; Shaw et al., 2018; Sukhbaatar et al., 2019a;b; Child et al., 2019).”
Improving Transformer Optimization Through Better Initialization (2020, background): “, 2020), recent research has focused on understanding and improving optimization in these models (Ahmed et al., 2017; Chen et al., 2018b; Nguyen & Salazar, 2019; Aji & Heafield, 2019; Liu et al., 2020).”