Deep Neural Machine Translation with Linear Associative Unit

By Mingxuan Wang, Zhengdong Lu, Jie Zhou, Qun Liu. 2017

54 citations recorded by the source corpus.

Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art Neural Machine Translation (NMT) with its capability in modeling complex functions and capturing complex linguistic structures. However NMT with deep architecture in its encoder or decoder RNNs often suffer from severe gradient diffusion due to the non-linear recurrent activations, which often makes the optimization much more difficult. To address this problem we propose a novel linear associative units (LAU) to reduce the gradient propagation path inside the recurrent unit. Different from conventional approaches (LSTM unit and GRU), LAUs uses linear associative connections between input and output of the recurrent unit, which allows unimpeded information flow through both space and time The model is quite simple, but it is surprisingly effective. Our empirical study on Chinese-English translation shows that our model with proper configuration can improve by 11.7 BLEU upon Groundhog and the best reported on results in the same setting. On WMT14 English-German task and a larger WMT14 English-French task, our model achieves comparable results with the state-of-the-art.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Lijun Wu
2. Tao Qin
3. Ankur Bapna
4. Felix Stahlberg
5. Xu Tan
6. Yingce Xia
7. Biao Zhang
8. Orhan Firat

Citations in the corpus

• Learning Deep Transformer Models for Machine Translation (2019, neutral): “Wang et al. (2018a) (post-norm, Base) 102.”
• Training Deeper Neural Machine Translation Models with Transparent Attention (2018, background): “…of capturing long range dependencies in recurrent architectures (Pascanu et al., 2012; Hochreiter et al., 2001), but resolving these deficiencies in Transformers or LSTM seq2seq models deeper than 8 layers is unfortunately under-explored (Wang et al., 2017; Barone et al., 2017; Devlin, 2017).”
• Towards Robust Neural Machine Translation (2018, background): “51 BLEU points over Wang et al. (2017) and up to +2 .”
• Unsupervised Neural Machine Translation with Weight Sharing (2018, methodology): “Similar to (Zhou et al., 2016; Wang et al., 2017), we utilize a self-attention network to process the input sequence in forward direction.”
• Multi-layer Representation Fusion for Neural Machine Translation (2018, unknown): “Another related study is Wang et al. (2017).”
• Depth Growing for Neural Machine Translation (2019, background): “Similar principles can be found in Wang et al. (2017); Wu et al.”
• Neural Machine Translation With GRU-Gated Attention Model (2020, methodology): ““ LAU ” AND “ F-F ” D ENOTE THE L INEAR A SSOCIATIVE U NIT AND THE F ASTFORWARD A RCHITECTURE P ROPOSED IN [37] AND [33], , where μ k is defined in (1). problem for NMT.”