Training Deeper Neural Machine Translation Models with Transparent Attention

By Ankur Bapna, M. Chen, Orhan Firat, Yuan Cao, Yonghui Wu. 2018

147 citations recorded by the source corpus.

While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT’14 English-German and WMT’15 Czech-English tasks for both architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “This is further exacerbated by the increasing model capacity of state-of-the-art NMT models (Shazeer et al., 2018; Bapna et al., 2018; Huang et al., 2018); full ﬁne-tuning is just too parameter inefﬁcient.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “…et al., 2017; Arora et al., 2018), trainability challenges including exploding and vanishing gradients (Glo-rot and Bengio, 2010; Balduzzi et al., 2017) and empirical approaches to counter these challenges (Bapna et al., 2018; Zhang et al., 2019) are critical to further scaling of model capacity.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Scaling Transformer architecture has been an exploratory research track recently [49, 50, 51].”
The NiuTrans Machine Translation Systems for WMT19 (2019, background): “Bapna et al. (2018) has shown that learning deeper networks is not easy for vanilla Transformer due to the gradient vanishing/exploding problem.”
Learning Deep Transformer Models for Machine Translation (2019, methodology): “L = 20, the vanilla Transformer failed to train, which is consistent with Bapna et al. (2018).”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “Transparent attention (Bapna et al., 2018) creates weighted residual connections along encoder depth to facilitate gradient flow.”
Deep Modular Co-Attention Networks for Visual Question Answering (2019, neutral): “Similar observations are also reported by [5].”
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020, neutral): “Deeper neural models are often capable of inducing more generalizable (‘abstract’) representations and capturing more complex dependencies and have shown encouraging performance on bilingual translation (Bapna et al., 2018; Zhang et al., 2019; Wang et al., 2019a).”
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, background): “Bapna et al. (2018) explored using learned linear combinations of encoder outputs as decoder inputs, while HAT concatenates the outputs without linear combinations, thus better preserving the low-level information.”
Very Deep Self-Attention Networks for End-to-End Speech Recognition (2019, background): “However, deep models in general suffer from either overfitting due to more complex architectures and optimization difficulty [17].”
AutoFormer: Searching Transformers for Visual Recognition (2021, result): “Recent works [3, 57] show that deep transformer is hard to train, which coincides with our observation in Fig.”
Depth Growing for Neural Machine Translation (2019, methodology): “Bapna et al. (2018) propose an enhancement to the attention mechanism to ease the optimization of models with deeper encoders.”
Learning Light-Weight Translation Models from Deep Transformer (2020, background): “For example, Bapna et al. (2018) and Wang et al. (2019) shortened the path from upper-level layers to lower-level layers so as to avoid gradient vanishing/exploding.”
Transformers in Time-Series Analysis: A Tutorial (2022, background): “attention-based operations, Transformers can correlate information from all elements in the sequence to each other in parallel without suffering from vanishing gradients as is the case with RNNs and their variants [9, 62, 92].”
A Survey of Transformers (2021, methodology): “Connectivity Realformer[51], Predictive Attention Transformer[143], Transparent Attention[8] Feedback Transformer [34]”
Robust Neural Machine Translation with Doubly Adversarial Inputs (2019, background): “Deeper neural networks with increased model capacities in NMT have also been explored and shown promising results ( Bapna et al., 2018).”