Learning to Encode Position for Transformer with Continuous Dynamical Model

By Xuanqing Liu, Hsiang-Fu Yu, Inderjit S. Dhillon, Cho-Jui Hsieh. 2020

141 citations recorded by the source corpus.

We introduce a new way of learning to encode position information for non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which contain inductive bias by loading the input tokens sequentially, non-recurrent models are less sensitive to position. The main reason is that position information among input units is not inherently encoded, i.e., the models are permutation equivalent; this problem justifies why all of the existing models are accompanied by a sinusoidal encoding/embedding layer at the input. However, this solution has clear limitations: the sinusoidal encoding is not flexible enough as it is manually designed and does not contain any learnable parameters, whereas the position embedding restricts the maximum length of input sequences. It is thus desirable to design a new position layer that contains learnable parameters to adjust to different datasets and different architectures. At the same time, we would also like the encodings to extrapolate in accordance with the variable length of inputs. In our proposed solution, we borrow from the recent Neural ODE approach, which may be viewed as a versatile continuous version of a ResNet. This model is capable of modeling many kinds of dynamical systems. We model the evolution of encoded results along position index by such a dynamical system, thereby overcoming the above limitations of existing methods. We evaluate our new position layers on a variety of neural machine translation and language understanding tasks, the experimental results show consistent improvements over the baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tianyang Lin
2. Yuxin Wang
3. Xiangyang Liu
4. Ulme Wennberg
5. Derya Soydaner
6. Bin Sheng
7. D. Feng
8. Xipeng Qiu

In Computer Vision and Pattern Recognition

1. Ulme Wennberg
2. Derya Soydaner
3. Bin Sheng
4. Ping Li
5. D. Feng
6. Xipeng Qiu
7. Tianyang Lin
8. Yuxin Wang

Citations in the corpus

• EAPT: Efficient Attention Pyramid Transformer for Image Processing (2021, methodology): “recursive encoding [17] adopts Neural ODE.”
• The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, methodology): “Liu et al. (2020) identified three important criteria for ideal position encoding: Approaches should be inductive , meaning that they can handle sequences and linguistic dependencies of arbitrary length, data-driven , meaning that positional dependencies are learned from data, and efficient in terms…”
• A Survey of Transformers (2021, methodology): “Later works find it beneficial to add position representations to inputs to each Transformer layer [2, 26, 45, 85].”
• Attention mechanism in neural networks: where it comes and where it goes (2022, background): “FLOw-bAsed TransformER (FLOATER) emphasizes the importance of position encoding in the Transformer, and models the position information via a continuous dynamical model [127].”