ViViT: A Video Vision Transformer

By Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, Cordelia Schmid. 2021

3,120 citations recorded by the source corpus.

We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatiotemporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “, 2020) but also computer vision (Dosovitskiy et al., 2020; Arnab et al., 2021), reinforcement 2”
Multiview Transformers for Video Recognition (2022, methodology): “We follow [3] and apply the same data augmentation and regularization schemes [13,29,65,83], which were used by [68] to train vision transformers more effectively.”
FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The self-supervised pretraining paradigm has significantly advanced the state of the art across various domains, from natural language processing [6, 17–19, 23, 24, 28, 30, 61, 68, 73, 83–85], to computer vision [2, 5, 8, 12, 31, 33, 37, 59, 75, 104, 117], to speech recognition [4, 22, 42, 67, 119] ”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, background): “[4, 6, 87] make attempts to apply Transformers on video recognition in a convolution-free fashion.”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “Comparison to state-of-the-art methods, including ViViT (Arnab et al., 2021), VideoSwin (Liu et al.”
Accelerating Large Language Model Decoding with Speculative Sampling (2023, background): “Scaling transformer models to 500B+ parameters has led to large performance improvements on many natural language, computer vision and reinforcement learning tasks (Arnab et al., 2021; Brown et al., 2020; Chowdhery et al., 2022; Dosovitskiy et al., 2020; Hoﬀmann et al., 2022; Rae et al., 2021).”
The Efficiency Misnomer (2021, methodology): “FLOPs: A widely used metric as the proxy for the computational cost of a model is the number of ﬂoating-point multiplication-and-addition operations (Johnson, 2018; Kim et al., 2021; Arnab et al., 2021; Tay et al., 2021a;b; Narayanan et al., 2021; Liu et al., 2021).”
Transformers in Vision: A Survey (2021, methodology): “A few exceptions include [10, 18, 147, 215] which obtain frame-level features also using the ViT based backbones.”
Segmenter: Transformer for Semantic Segmentation (2021, neutral): “Concurrent work extends this work to video classi-ﬁcation [2, 6] and semantic segmentation [35, 67].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “Since then, the pure Transformer design has been adopted to various vision tasks including semantic segmentation [114], point cloud classification [113], action recoginition [10, 83, 6].”
Multiscale Vision Transformers (2021, methodology): “Concurrent vision-transformer based methods [78,6,1] require over 5× more computation and large-scale external pre-training on ImageNet-21K (IN-21K), to achieve equivalent MViT accuracy.”
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval (2021, methodology): “Such a 2D linear ignores the temporal information among frames. b) Therefore, we investigate a 3D linear projection, similar to (Arnab et al., 2021), to enhance temporal feature extraction.”
Video Diffusion Models (2022, background): “The use of factorized space-time attention is known to be a good choice in video transformers for its computational efﬁciency [2, 5, 21].”
XCiT: Cross-Covariance Image Transformers (2021, background): “Promising results have been obtained for other vision tasks, including image retrieval [22], object detection and semantic segmentation [44, 70, 81, 83], as well as video understanding [2, 7, 23].”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “Third, for larger models, we use stochastic depth regularisation [17,1] with probability of 0.”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Inspired by the success of Transformers in NLP [20,76], the Vision Transformer (ViT) [22] was recently proposed and has been successfully applied to multiple computer vision tasks, including image classification [44, 72, 73, 86], object detection [44, 79, 88], semantic segmentation [44, 82, 91] and ”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “MoViNet-A5 and MoViNet-A6 outperform several state-of-the-art video networks, including recent Transformer models like ViViT [2] and TimeSformer [5] (see the last 6 rows of Table 2).”
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, methodology): “In high-level vision, Transformer has been widely applied in image classification [7, 19, 25, 27, 49, 65], object detection [6, 22, 23, 60, 86, 93], semantic segmentation [16, 69,76,77,90,91], human pose estimation [14,15,32,37,42, 43, 53, 79, 89], etc.”