Exploiting Deep Representations for Neural Machine Translation

By Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, T. Zhang. 2018

96 citations recorded by the source corpus.

Advanced neural machine translation (NMT) models generally implement encoder and decoder as multiple layers, which allows systems to model complex functions and capture complicated linguistic structures. However, only the top layers of encoder and decoder are leveraged in the subsequent process, which misses the opportunity to exploit the useful information embedded in other layers. In this work, we propose to simultaneously expose all of these signals with layer aggregation and multi-layer attention mechanisms. In addition, we introduce an auxiliary regularization term to encourage different layers to capture diverse information. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation data demonstrate the effectiveness and universality of the proposed approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Chenguang Zhu
6. Jianfeng Gao
7. Shuohang Wang
8. Nanyun Peng

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Lijuan Wang
3. Zicheng Liu
4. Jianfeng Wang
5. Jianfeng Gao
6. Luc Van Gool
7. Shuohang Wang
8. Cordelia Schmid

In Natural Language Processing

1. Tong Xiao
2. Jingbo Zhu
3. Bei Li
4. Qiang Wang
5. Chenguang Zhu
6. Atsushi Fujita
7. Longyue Wang
8. Michael Zeng

Citations in the corpus

• Learning Deep Transformer Models for Machine Translation (2019, methodology): “, 2017) and Transformer (Dou et al., 2018).”
• Convolutional Self-Attention Networks (2019, background): “Inspired by recent successes on fusing information across layers (Dou et al., 2018, 2019), Li et al. (2019) proposed to aggregate information captured by different attention heads.”
• Modeling Recurrence for Transformer (2019, background): “Concerning multi-layer SAN encoder, Dou et al. (2018, 2019) and Wang et al. (2018) propose to aggregate the multi-layer representations, and De-hghani et al. (2019) recurrently refine these representations.”
• Rethinking Skip Connection with Layer Normalization (2020, unknown): “0 Deep Representations (Dou et al., 2018) 356M 29.”
• An Empirical Study of Training End-to-End Vision-and-Language Transformers (2021, background): “Aggregating the features at different layers has demonstrated to be helpful in both vision [6, 17] and language [2, 5].”
• Balancing Cost and Benefit with Tied-Multi Transformers (2020, background): “Belinkov et al. (2017) and Dou et al. (2018) attempted to identify which layer can generate useful representations for different natural language processing tasks.”