Relative Positional Encoding for Transformers with Linear Complexity

By Antoine Liutkus, Ondřej Cífka, Shih-Lun Wu, Umut Simsekli, Yi-Hsuan Yang, Gaël Richard. 2021

64 citations recorded by the source corpus.

Recent advances in Transformer models allow for unprecedented sequence lengths, due to linear space and time complexity. In the meantime, relative positional encoding (RPE) was proposed as beneficial for classical Transformers and consists in exploiting lags instead of absolute positions for inference. Still, RPE is not available for the recent linear-variants of the Transformer, because it requires the explicit computation of the attention matrix, which is precisely what is avoided by such methods. In this paper, we bridge this gap and present Stochastic Positional Encoding as a way to generate PE that can be used as a replacement to the classical additive (sinusoidal) PE and provably behaves like RPE. The main theoretical contribution is to make a connection between positional encoding and cross-covariance structures of correlated Gaussian processes. We illustrate the performance of our approach on the Long-Range Arena benchmark and on music generation.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhen Qin
2. Junjie Yan
3. Yunfeng Liu
4. Yiran Zhong
5. Dongxu Li
6. Lingpeng Kong
7. Jianlin Su
8. Martin Wistuba

In Model Architectures

1. Zhen Qin
2. Junjie Yan
3. Yiran Zhong
4. Dongxu Li
5. Lingpeng Kong
6. Jianlin Su

In Signal Processing

1. Junjie Yan
2. Lingpeng Kong

Citations in the corpus

• RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, background): “[21] proposed to view positional attention as covariance and draw samples from centered random processes and [13] proposed the relative position embeddings for kernelizable attention.”
• cosFormer: Rethinking Softmax in Attention (2022, methodology): “For Stochastic Positional Encoding (Liutkus et al., 2021), they use a sampling strategy to approximate the softmax, and introduce relative positional encoding to linear transformers.”