Depth Growing for Neural Machine Translation

By Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, J. Lai, Tie-Yan Liu. 2019

45 citations recorded by the source corpus.

While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of the neural machine translation (NMT) models for better translation quality remains a challenging problem. Directly stacking more blocks to the NMT model results in no improvement and even drop in performance. In this work, we propose an effective two-stage approach with three specially designed components to construct deeper NMT models, which result in significant improvements over the strong Transformer baselines on WMT14 English\toGerman and English\toFrench translation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Edouard Grave
2. Angela Fan
3. Armand Joulin
4. Luke S. Zettlemoyer
5. Jason Aaron Edward Weston
6. Mohit Bansal
7. Kyunghyun Cho
8. Douwe Kiela

In Computer Vision and Pattern Recognition

1. Edouard Grave
2. Armand Joulin
3. Angela Fan
4. Luc Van Gool
5. Dacheng Tao
6. Trevor Darrell
7. Jitendra Malik
8. Xiaogang Wang

In Model Architectures

1. Edouard Grave
2. Armand Joulin
3. Angela Fan
4. Chunliang Zhang
5. Tong Xiao
6. Heng Yu
7. Jingbo Zhu
8. Weihua Luo

In Machine Translation

1. Angela Fan
2. Armand Joulin
3. Edouard Grave

Citations in the corpus

• Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “Following Wu et al. (2019), we use the Transformer base architecture with 6 encoder layers and 6 decoder layers.”
• Learning Light-Weight Translation Models from Deep Transformer (2020, methodology): “We adopted the compound split strategy for En-De, which was a common post-processing step used in previous work (Vaswani et al. 2017; Wang et al. 2019; Wu et al. 2019b).”
• Towards Enhancing Faithfulness for Neural Machine Translation (2020, background): “Neural machine translation (NMT) based on the encoder-decoder framework (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014; Luong et al., 2015b) has obtained state-of-the-art performance on many language pairs (Wu et al., 2019; Wei et al., 2020).”