Focal Attention for Long-Range Interactions in Vision Transformers

By Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, Jianfeng Gao. 2021

214 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Jianwei Yang
2. Jianfeng Gao
3. Pengchuan Zhang
4. Lu Yuan
5. Bin Xiao
6. Chunyuan Li
7. Zhe Gan
8. Dacheng Tao

Citations in the corpus

• Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “With this goal, numerous works have pushed the image recognition performance from different directions, such as data scale from MNIST [36] to ImageNet1K [10], model architectures from convolutional neural networks (CNNs) [23, 26, 34, 35, 40, 52, 53] to Transformers [15, 43, 57, 61, 64, 67, 72], and ”
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone (2024, unknown): “We followed the standard training recipe of previous efforts [11, 21, 37] to allow for a comparable analysis of performance across different models.”
• Vision Transformers for Single Image Dehazing (2022, methodology): “Subsequently, many modified architectures [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41] have been proposed, and vision Transformer is challenging the dominance of CNNs in high-level vision tasks.”
• Illuminating protein space with a programmable generative model (2022, background): “Distant-dependent density of context has also been explored in multiresolution attention for Vision transformers [Yang et al., 2021] and in dilated convolutional neural networks [van den Oord et al.”
• Neighborhood Attention Transformer (2022, methodology): “[34] proposed Focal Attention, a windowbased mechanism that limits fine-grained attention to locally surrounding tokens, while applying coarse-grained attention to tokens that are further away.”
• Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “There are some methods aiming to reduce the computational complexity of self-attention or improve its ability of modeling visual dependencies (Song, 2021; Yang et al., 2021; Rao et al., 2021; Choromanski et al., 2020).”