Factorized Attention: Self-Attention with Linear Complexities

By Zhuoran Shen, Mingyuan Zhang, Shuai Yi, Junjie Yan, Haiyu Zhao. 2018

13 citations recorded by the source corpus.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Tri Dao
2. Zihang Dai
3. Adrian Weller
4. Valerii Likhosherstov
5. Xingyou Song
6. Jared Davis
7. Dale Schuurmans
8. Bo Dai

In Efficiency & Compression

1. Adrian Weller
2. Valerii Likhosherstov
3. Krzysztof Marcin Choromanski
4. Jared Davis
5. Xingyou Song
6. Zihang Dai
7. Bo Dai
8. Mengjiao Yang

Citations in the corpus

• Rethinking Attention with Performers (2020, background): “There is also a long line of research on using dense attention matrices, but defined by low-rank kernels substituting softmax (Katharopoulos et al., 2020; Shen et al., 2018).”
• Combiner: Full Attention Transformer with Sparse Computation Cost (2021, background): “Alternatively, another line of research has tried to achieve scalability with an explicit low-rank assumption [23, 24] on the attention matrix or by using explicit feature maps of some kernels [25].”