Video Swin Transformer

By Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, Han Hu. 2021

2,180 citations recorded by the source corpus.

The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchmarks. These video models are all built on Transformer layers that globally connect patches across the spatial and temporal dimensions. In this paper, we instead advocate an inductive bias of locality in video Transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization. The locality of the proposed video architecture is realized by adapting the Swin Transformer designed for the image domain, while continuing to leverage the power of pre-trained image models. Our approach achieves state-of-the-art accuracy on a broad range of video recognition benchmarks, including on action recognition (84.9 top-l accuracy on Kinetics-400 and 85.9 top-l accuracy on Kinetics-600 with ~20× less pre-training data and ~3× smaller model size) and temporal modeling (69.6 top-l accuracy on Something-Something v2).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

All in One: Exploring Unified Video-Language Pre-Training (2022, background): “Several classic ideas including sparse sampling [37], 3Dtype operations [6, 26] are proposed for temporal modeling in both convolution and Transformer architectures.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “We initialize our Video Swin Transformer with VideoSwin-Base [22], pre-trained on Kinetics-400 [37].”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Inspired by the success of Transformers in NLP [20,76], the Vision Transformer (ViT) [22] was recently proposed and has been successfully applied to multiple computer vision tasks, including image classification [44, 72, 73, 86], object detection [44, 79, 88], semantic segmentation [44, 82, 91] and ”
Dilated Neighborhood Attention Transformer (2022, background): “It also inspired other works that extended it to different tasks beyond the ones explored in the paper, such as generation [48], restoration [22], masked image modeling [45], video action recognition [28], and more.”
Masked Feature Prediction for Self-Supervised Visual Pre-Training (2021, neutral): “9% top-1 Swin-L↑384 [48] with IN-21K and 84.”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Also, our dual-path transformer encoder achieves a better trade-off than the vanilla 3D convolution and the 3D windowed attention proposed in [35].”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, methodology): “To attain such local-global properties efficiently, we propose a sparse 3D attention model called fused axial attention (FAX), which performs both local window-based attention and sparse global interactions, inspired by [32, 48, 49].”
BEVT: BERT Pretraining of Video Transformers (2021, methodology): “In particular, Video Swin Transformer [37] follows the design of Swin Transformer [36] and is a hierarchical architecture consisting of four stages.”
Egocentric Video-Language Pretraining (2022, background): “, the video backbone can be replaced by SlowFast [34] or Video Swin [35]).”
UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, background): “capacity of learning global dependency in both images [23], [60] and videos [1], [3], [61].”
Vision Transformer Adapter for Dense Predictions (2022, background): “With different embedding layers, such as patch embedding [22], 3D patch embedding [53], and token embedding [73], vanilla transformers such as ViT are able to process multi-modal data in terms of image, video, and text.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, methodology): “For visual feature extraction, instead of utilizing videobased foundational models such as ViViT [3] or VideoSwin [51], we simply use image-based model to get framewise feature in the form of tokens.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, background): “3.3.”
Video Graph Transformer for Video Question Answering (2022, background): “Current video encoders are either 2D neural networks (CNNs [18,45] or Transformers [13]) operated over sparse frames or 3D neural networks [5,37,61] operated over short video segments.”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, background): “Following this success, several efforts explore the potential of ViT by improving training strategies [3, 4, 5], introducing architecture changes [6, 7], redesigning attention mechanisms [8, 9], and elevating the performance of various vision tasks such as classification [10, 11, 12], segmentation [”
ActionFormer: Localizing Moments of Actions with Transformers (2022, background): “Transformer have also been explored in object detection [11, 17, 69, 87], semantic segmentation [15, 71, 73], and more recently video representation learning [3,23,49].”
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation (2021, background): “More recently, it has achieved great success on various computer vision tasks, e.g ., image classiﬁcation [14,35,53], action recognition [2, 36], object detection [4, 35, 67], and semantic segmentation [35,52,65].”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, methodology): “VideoMAE utilizes the plain ViT [27] architecture of joint space-time attention mechanism [2, 57] and an extremely high proportion of masking ratio (i.”