Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel

By Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-philippe Morency, Ruslan Salakhutdinov. 2019

333 citations recorded by the source corpus.

Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. At the core of the Transformer is the attention mechanism, which concurrently processes all inputs in the streams. In this paper, we present a new formulation of attention via the lens of the kernel. To be more precise, we realize that the attention can be seen as applying kernel smoother over the inputs with the kernel scores being the similarities between inputs. This new formulation gives us a better way to understand individual components of the Transformer’s attention, such as the better way to integrate the positional embedding. Another important advantage of our kernel-based formulation is that it paves the way to a larger space of composing Transformer’s attention. As an example, we propose a new variant of Transformer’s attention which models the input as a product of symmetric kernels. This approach achieves competitive performance to the current state of the art model with less computation. In our experiments, we empirically study different kernel construction strategies on two widely used tasks: neural machine translation and sequence prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Rethinking Attention with Performers (2020, neutral): “This is similar to (Tsai et al., 2019) which also experiments with various attention kernels for natural language.”
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2020, methodology): “Tsai et al. (2019) proposed a kernel-based formulation of attention in transformers which considers attention as applying a kernel smoother over the inputs with the kernel scores being the similarity between inputs.”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, background): “Under the general name of relative attention, there have been various variants in literature [30, 38, 39, 34, 40, 31].”
Random Feature Attention (2021, background): “This supplements the exploration of alternatives to softmax in attention (Tsai et al., 2019; Gao et al., 2019).”
Relative Positional Encoding for Transformers with Linear Complexity (2021, positive): “In Irie et al. (2019) and Tsai et al. (2019), it is found that removing APE altogether in the causal decoder part of Transformer-based architectures leads to comparable/better performance.”
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2020, methodology): “Following the formulation in (Tsai et al. 2019), the i -th query’s attention is deﬁned as a kernel smoother in a probability form: where p ( k j | q i ) = k ( q i , k j ) l k ( q i , k l ) and k ( q i , k j ) selects the asymmetric exponential kernel exp( The self-attention combines the values and…”
The Lipschitz Constant of Self-Attention (2020, background): “Another interesting direction for future work would be to analyse different variants of self-attention based on kernels other than dot-product and L2, as (Tsai et al., 2019) do from an experimental perspective, for which we believe the mathematical tools developed in this paper may aid the analysis.”
The Impact of Positional Encoding on Length Generalization in Transformers (2023, background): “However, decoder-only Transformers with causal attention mask are not permutation invariant and can model sequences even without explicit position information (Tsai et al., 2019).”
cosFormer: Rethinking Softmax in Attention (2022, background): “…into the soft-max attention, we ﬁnd two key properties that affect its empirical performance: (i) elements in the attention matrix are non-negative (Tsai et al., 2019; Katharopoulos et al., 2020); (ii) the non-linear re-weighting scheme acts as a stabilizer for the attention weights (Titsias,…”
A Survey of Transformers (2021, background): “It is worth noticing that masked selfattention is not permutation equivariant [133].”
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images (2020, methodology): “The formulae and mathematical proofs can be found in [54].”
Choose a Transformer: Fourier or Galerkin (2021, methodology): “The kernel interpretation in [84] inspires us to reformulate the attention using the Galerkin projection.”
ETSformer: Exponential Smoothing Transformers for Time-series Forecasting (2022, background): “Vanilla self-attention can be regarded as a weighted combination of an input sequence, where the weights are normalized alignment scores measuring the similarity between input contents [32].”
Quasi Linear Kernel Attention with Infinite Capacity (2026, unknown): “In this paper, we focus on exact subquadratic attention, where the structure of the kernel Φ allows the computation of (1) in quasi linear time (Tsai et al., 2019).”