CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings

By Tatiana Likhomanenko, Qiantong Xu, Ronan Collobert, Gabriel Synnaeve, Alex Rogozhnikov. 2021

73 citations recorded by the source corpus.

Without positional information, attention-based Transformer neural networks are permutation-invariant. Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional information. Absolute positional embeddings are simple to implement, but suffer from generalization issues when evaluating on sequences longer than seen at training time. Relative positions are more robust to input length change, but are more complex to implement and yield inferior model throughput due to extra computational and memory costs. In this paper, we propose an augmentation-based approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute (simplicity and speed) and relative positional embeddings (better generalization). In addition, our empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance as well as increased stability with respect to training hyper-parameters.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021, methodology): “, 2021), image, speech recognition, and machine translation models (Likhomanenko et al., 2021), and protein structure prediction (Jumper et al.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “There is a great need for more novel position encoding mechanisms like CAPE [288] which combines the advantages in both absolute and relative position embeddings.”
DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “This is also similar to UniViT training from Likhomanenko et al. (2021) and FlexiViT training from Beyer et al. (2023).”
The Impact of Positional Encoding on Length Generalization in Transformers (2023, background): “The length generalization problem has been reported even in pretrained Transformers such as T5 (Furrer et al., 2020) and LaMDA (Anil et al., 2022).”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, methodology): “Further research could explore how diﬀerent types of positional encodings (such as learned embeddings, sinusoidal embeddings, or CAPE [Likhomanenko et al., 2021]) and diﬀerent attention mechanisms (such as saturated attention [Merrill et al., 2021]) could help the models overcome even these…”