Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

By Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, Jianfeng Gao. 2021

395 citations recorded by the source corpus.

This paper presents a new Vision Transformer (ViT) architecture Multi-Scale Vision Longformer, which significantly enhances the ViT of [12] for encoding high-resolution images using two techniques. The first is the multi-scale model structure, which provides image encodings at multiple scales with manageable computational cost. The second is the attention mechanism of Vision Long-former, which is a variant of Longformer [3], originally developed for natural language processing, and achieves a linear complexity w.r.t. the number of input tokens. A comprehensive empirical study shows that the new ViT significantly outperforms several strong baselines, including the existing ViT models and their ResNet counterparts, and the Pyramid Vision Transformer from a concurrent work [47], on a range of vision tasks, including image classification, object detection, and segmentation. The models and source code are released at https://github.com/microsoft/vision-longformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Florence: A New Foundation Model for Computer Vision (2021, methodology): “For the image encoder, we chose hierarchical Vision Transformers (e.g. , Swin (Liu et al., 2021a), CvT (Wu et al., 2021), Vision Longformer (Zhang et al., 2021a), Focal Transformer (Yang et al., 2021), and CSwin (Dong et al., 2021)).”
Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “With this goal, numerous works have pushed the image recognition performance from different directions, such as data scale from MNIST [36] to ImageNet1K [10], model architectures from convolutional neural networks (CNNs) [23, 26, 34, 35, 40, 52, 53] to Transformers [15, 43, 57, 61, 64, 67, 72], and ”
Grounded Language-Image Pre-training (2021, methodology): “We ﬁrst choose the SoTA detection model Dynamic Head (Dy-Head) [10] based on the Swin-Tiny Transformer backbone [39] as the base object detection mode.”
EDTER: Edge Detection with Transformer (2022, background): “This architecture brings direct inspiration to other computer vision tasks [32, 38, 55, 68, 72].”
Transformers in Vision: A Survey (2021, background): “Multi-Scale Vision Longformer [243] also considers a local context in self-attention, but employs the eicient Longformer [15] design for self-attention.”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “Since these works, numerous vision Transformer variants have been proposed to improve the accuracy at relatively small scale [14,21,34,42,63,68,71,75,77,78,82].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “When we increase the window size from 7 to 14 to match the settings in ViL-Small [80], the performance can be further improved to 82.”
XCiT: Cross-Covariance Image Transformers (2021, methodology): “Moreover, for dense prediction tasks such as object detection and image segmentation, our models outperform popular ResNet [28] backbones as well as the recent transformer-based models [44, 72, 82].”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Inspired by the success of Transformers in NLP [20,76], the Vision Transformer (ViT) [22] was recently proposed and has been successfully applied to multiple computer vision tasks, including image classification [44, 72, 73, 86], object detection [44, 79, 88], semantic segmentation [44, 82, 91] and ”
Efficient Self-supervised Vision Transformers for Representation Learning (2021, background): “(ii) Vision Longformer (ViL) [70]: Features in each local”
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer (2021, background): “…CAIT (Touvron et al., 2021b), DeepViT (Zhou et al., 2021), CeiT (Yuan et al., 2021a), CrossViT (Chen et al., 2021a), LocalViT (Li et al., 2021), PiT (Heo et al., 2021), ConViT (d’Ascoli et al., 2021), ViL (Zhang et al., 2021), BoTNet (Srinivas et al., 2021), and Mobile-former (Chen et al., 2021b)).”
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows (2021, background): “Rather than concentrating on one special task, some recent works [30, 47, 56] try to design a general vision Transformer backbone for general-purpose vision tasks.”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “While [37, 58, 67, 71] has a simple pyramid structure (fine-to-coarse), XCiT [17] has a single-stage structure as ViT [16].”
Aggregating Nested Transformers (2021, background): “…mechanism [41, 51] attract wide interests in recent computer vision community, with explorations of more suitable architectural design that can learn visual representation effectively, such as injecting convnet elements [33, 46, 55], building local or hierarchical structures [54, 62].”
A Survey of Visual Transformers (2021, methodology): “Local Enhanced Global TNT [52], Twins [53], Focal Transformer [54], ViL [55]”
Auto-scaling Vision Transformers without Training (2022, background): “…representation learning, recently works try to train deeper ViTs (Touvron et al., 2021; Zhou et al., 2021), incorporate convolutions (Wu et al., 2021; d’Ascoli et al., 2021; Yuan et al., 2021a), and design multi-scale feature extractions (Chen et al., 2021b; Zhang et al., 2021; Wang et al., 2021).”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, methodology): “[35], [41], [51], [53], [70], [93], [114] is beneficial for dense prediction tasks such as object detection, instance and semantic segmentation, recent works [21], [23], [32], [38], [42], [54], [58], [66], [77], [78], [95], [104] introduce multi-scale modeling capability by using a pyramid structure”
End-to-End Instance Edge Detection (2022, methodology): “To address this, researchers have proposed methods [68,60,44] that reduce both computation and memory complexity, allowing the transformer to perform dense prediction tasks [63,12,54,59].”