Analysis of Positional Encodings for Neural Machine Translation

By Jan Rosendahl, Viet Anh Khoa Tran, Weiyue Wang, Hermann Ney. 2019

21 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Tatiana Likhomanenko
2. Qiantong Xu
3. Gabriel Synnaeve
4. Ronan Collobert
5. Alexei Baevski
6. Wei-Ning Hsu
7. Yoshua Bengio
8. Jun Suzuki

Citations in the corpus

• Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “Since the literature suggests that RPE improves generalization performance (Shaw et al., 2018; Zhou et al., 2019; Rosendahl et al., 2019), we display validation cross-entropy computed with teacher forcing (Williams & Zipser, 1989) in Figure 3, as a function of the target token position.”
• SHAPE: Shifted Absolute Position Embedding for Transformers (2021, background): “RPE outperforms APE on out-of-distribution data in terms of sequence length owing to its innate shift invariance (Rosendahl et al., 2019; Neishi and Yoshinaga, 2019; Narang et al., 2021; Wang et al., 2021).”
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings (2021, background): “A number of approaches were thus proposed to encode relative positions in an explicit form [44, 11, 26, 42, 27], leading to improvements in modeling long sequences.”