Multi-layer Representation Fusion for Neural Machine Translation
By Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, Jingbo Zhu. 2018
55 citations recorded by the source corpus.
Neural machine translation systems require a number of stacked layers for deep models. But the prediction depends on the sentence representation of the top-most layer with no access to low-level representations. This makes it more difficult to train the model and poses a risk of information loss to prediction. In this paper, we propose a multi-layer representation fusion (MLRF) approach to fusing stacked layers. In particular, we design three fusion functions to learn a better representation from the stack. Experimental results show that our approach yields improvements of 0.92 and 0.56 BLEU points over the strong Transformer baseline on IWSLT German-English and NIST Chinese-English MT tasks respectively. The result is new state-of-the-art in German-English translation.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Natural Language Processing
Citations in the corpus
- Learning Deep Transformer Models for Machine Translation (2019, methodology): “We refer the reader to Wang et al. (2018a) for the details.”
- Modeling Recurrence for Transformer (2019, background): “Concerning multi-layer SAN encoder, Dou et al. (2018, 2019) and Wang et al. (2018) propose to aggregate the multi-layer representations, and De-hghani et al. (2019) recurrently refine these representations.”
- Balancing Cost and Benefit with Tied-Multi Transformers (2020, background): “Wang et al. (2018) fused hidden representations of multiple layers in order to improve the translation quality.”