Learning Deep Transformer Models for Machine Translation

By Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, Lidia S. Chao. 2019

775 citations recorded by the source corpus.

Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT’16 English-German and NIST OpenMT’12 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Layernorm (LN) is applied before every block, and residual connections after every block (Wang et al., 2019; Baevski & Auli, 2019).”
Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “We also evaluate the model by ﬁnetuning on tasks from GLUE (Wang et al., 2019a).”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Scaling Transformer architecture has been an exploratory research track recently [49, 50, 51].”
The NiuTrans Machine Translation Systems for WMT19 (2019, methodology): “We proposed four novel deep Transformer architectures based on (Wang et al., 2019) as our baseline, which surpass the standard Transformer-Big significantly in terms of both translation quality and convergence speed.”
Early Convolutions Help Transformers See Better (2021, methodology): “The resulting patch embeddings (plus positional embeddings and a learned classification token embedding) are processed by a standard transformer encoder [43, 44] followed by a classification head.”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, methodology): “3On the much easier QQP test set, RULEBERT achieved 0.89 accuracy after one epoch, and 0.91 after three epochs.”
Transformers as Soft Reasoners over Language (2020, background): “, 2019; Richardson and Sabharwal, 2019] or can learn to emulate other algorithms, including for semantic parsing [He and Choi, 2019], machine translation [Wang et al., 2019], integration [Lample and Charton, 2019], and math [Saxton et al.”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “We use prenorm residual units [21, 22] with dropout which helps training and regularizing deeper models.”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “As typically used in sequence-to-sequence transduction tasks [4, 5, 6, 7, 8], Transformerbased models attend over encoder features using decoder features, implying that the decoding has to be done in a label-synchronous way, thereby posing a challenge for streaming speech recognition applications.”
Reducing Transformer Depth on Demand with Structured Dropout (2019, background): “…translation, WikiText-103 (Merity et al., 2016) for language modeling, CNNDailymail (Hermann et al., 2015) for abstractive summarization, ELI5 (Fan et al., 2017) for long form question answering, and several natural language understanding tasks (Wang et al., 2019) for sentence representation.”
On Layer Normalization in the Transformer Architecture (2020, methodology): “In particular, we study another variant, the Trans-former with Pre-Layer Normalization (Pre-LN) (Baevski & Auli, 2018; Child et al., 2019; Wang et al., 2019).”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “Transformer models [6], [77], [78] are ﬁrst proposed to handle natural language processing tasks, e.g., text classi-ﬁcation [79], [80], question answering [81] and machine translation [82].”
Transformers without Tears: Improving the Normalization of Self-Attention (2019, methodology): “Surprisingly, in this task POSTNORM works notably better than PRENORM; one observes similar behavior in [6].”
Pre-Trained Image Processing Transformer (2020, methodology): “The seminal work Transformers [61] have been widely used in many natural language processing (NLP) tasks, such as translation [64] and questionanswering [58].”
PRover: Proof Generation for Interpretable Reasoning over Rules (2020, background): “, 2019) or can be fine-tuned for tasks like semantic parsing (He and Choi, 2020), translation (Wang et al., 2019), symbolic integration (Lample and Charton, 2020) and mathematics (Saxton et al.”
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, background): “Recently, a research trend is to modify the Transformer to improve the performance (Chen et al., 2018; Wu et al., 2019b; Sukhbaatar et al., 2019; Wang et al., 2019).”
FlauBERT: Unsupervised Language Model Pre-training for French (2019, methodology): “The authors of GLUE have therefore introduced SuperGLUE (Wang et al., 2019a): a new benchmark built on the principles of GLUE, including more challenging and diverse set of tasks.”
On Learning Universal Representations Across Languages (2020, methodology): “GLUE We evaluate the English performance of our model on the GLUE benchmark (Wang et al., 2019a) which gathers multiple classication tasks, including single-sentence tasks (e.g., SST-2 (Socher et al., 2013)), similarity and paraphrase tasks (MRPC (Dolan & Brockett, 2005), STS-B (Cer et al., 2017),…”