On the Relation between Position Information and Sentence Length in Neural Machine Translation
By Masato Neishi, Naoki Yoshinaga. 2019
47 citations recorded by the source corpus.
Long sentences have been one of the major challenges in neural machine translation (NMT). Although some approaches such as the attention mechanism have partially remedied the problem, we found that the current standard NMT model, Transformer, has difficulty in translating long sentences compared to the former standard, Recurrent Neural Network (RNN)-based model. One of the key differences of these NMT models is how the model handles position information which is essential to process sequential data. In this study, we focus on the position information type of NMT models, and hypothesize that relative position is better than absolute position. To examine the hypothesis, we propose RNN-Transformer which replaces positional encoding layer of Transformer by RNN, and then compare RNN-based model and four variants of Transformer. Experiments on ASPEC English-to-Japanese and WMT2014 English-to-German translation tasks demonstrate that relative position helps translating sentences longer than those in the training data. Further experiments on length-controlled training data reveal that absolute position actually causes overfitting to the sentence length.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Kentaro Inui
- Shun Kiyono
- Jun Suzuki
- Dan Ofer
- Michal Linial
- Sosuke Kobayashi
- Nadav Brandes
- Sergey I. Nikolenko
Citations in the corpus
- Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “Neishi & Yoshinaga (2019) builds on this view, but explicitly describes the idea for the first time.”
- ProteinBERT: a universal deep-learning model of protein sequence and function (2021, result): “This good generality across sequence lengths is also achieved by avoiding positional embeddings used in the standard version of BERT which, in accordance with previous reports [42] and our experimentation, do not always generalize well to sequence lengths longer than those present in the training da”
- SHAPE: Shifted Absolute Position Embedding for Transformers (2021, methodology): “On E XTRAPOLATE , RPE (29.86) outperforms APE (29.22) by approximately 0.6 BLEU points on the test set; this is consistent with the result re-ported by Neishi and Yoshinaga (2019).”