Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View

By Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, Tie-Yan Liu. 2019

234 citations recorded by the source corpus.

The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible codes and pretrained models can be found at this https URL

Subfields: Artificial Intelligence, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tie-Yan Liu
2. Xu Tan
3. Jianfeng Gao
4. Tao Qin
5. Quoc V. Le
6. Yu Zhang
7. Mohit Bansal
8. Furu Wei

In Model Architectures

1. Xu Tan
2. Tie-Yan Liu
3. Lei He
4. Sheng Zhao
5. Jianfeng Gao
6. Noah A. Smith
7. Yu Zhang
8. Yonghui Wu

In Scaling & Learning Theory

1. Noah A. Smith
2. Jiawei Han
3. Jianfeng Gao
4. Tie-Yan Liu
5. Liwei Wang
6. Huishuai Zhang
7. Ofir Press
8. Quanquan Gu

Citations in the corpus

• Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “Inspired by [17, 18], we introduce a novel combination of self-attention and convolution, sandwiched between a pair feed forward modules, as illustrated in Fig 1.”
• On Layer Normalization in the Transformer Architecture (2020, background): “However, the learning rate warm-up stage is essential and critical when optimizing the Transformer models in a majority of scenarios (Vaswani et al., 2017; Devlin et al., 2018; Dai et al., 2019; Radford et al., 2019; Lu et al., 2019).”
• Improving Transformer Models by Reordering their Sublayers (2019, background): “Recently, Lu et al. (2019) introduced a new transformer ordering, where instead of stacking layers of the form sf (as in the vanilla interleaved transformer), they stack layers of the form fsf.”
• DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021 (2021, methodology): “Third, we replace the linear layers in feed forward-modules of Macron [26] structure with convolution layers, which results in better prosody and audio fidelity.”
• Understanding the Difficulty of Training Transformers (2020, methodology): “For Model training, we use RAdam as the optimizer (Liu et al., 2020a) and adopt almost all hyper-parameter settings from Lu et al. (2020).”
• A Survey of Transformers (2021, methodology): “ET[123], Macaron Transformer[89], Sandwich Transformer[99], MAN[35], DARTSformer[167]”