Video Transformer Network

By Daniel Neimark, Omri Bar, Maya Zohar, Dotan Asselmann. 2021

500 citations recorded by the source corpus.

This paper presents VTN, a transformer-based framework for video recognition. Inspired by recent developments in vision transformers, we ditch the standard approach in video action recognition that relies on 3D ConvNets and introduce a method that classifies actions by attending to the entire video sequence information. Our approach is generic and builds on top of any given 2D spatial network. In terms of wall runtime, it trains 16.1× faster and runs 5.1× faster during inference while maintaining competitive accuracy compared to other state-of-the-art methods. It enables whole video analysis, via a single end-to-end pass, while requiring 1.5× fewer GFLOPs. We report competitive results on Kinetics-400 and Moments in Time benchmarks and present an ablation study of VTN properties and the trade-off between accuracy and inference speed. We hope our approach will serve as a new baseline and start a fresh line of research in the video recognition domain. Code and models are available at: https://github.com/bomri/SlowFast/blob/master/projects/vtn/README.md.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The self-supervised pretraining paradigm has significantly advanced the state of the art across various domains, from natural language processing [6, 17–19, 23, 24, 28, 30, 61, 68, 73, 83–85], to computer vision [2, 5, 8, 12, 31, 33, 37, 59, 75, 104, 117], to speech recognition [4, 22, 42, 67, 119] ”
Transformers in Vision: A Survey (2021, methodology): “A few exceptions include [209], [211]–[213] which obtain framelevel features also using the ViT based backbones.”
Multiscale Vision Transformers (2021, methodology): “Concurrent vision-transformer based methods [78,6,1] require over 5× more computation and large-scale external pre-training on ImageNet-21K (IN-21K), to achieve equivalent MViT accuracy.”
ViViT: A Video Vision Transformer (2021, methodology): “This architecture corresponds to a “late fusion” [34, 56, 72, 46] of temporal information, and the initial spatial encoder is identical to the one used for image classi-ﬁcation.”
ImageNet-21K Pretraining for the Masses (2021, background): “Many papers used these pretrained models for downstream tasks ( [55, 35, 31, 1] for example).”
DeiT III: Revenge of the ViT (2022, unknown): “[55], is now used as an alternative to convnets in many tasks: image classification [13, 48], detection [3, 31], semantic segmentation [2, 31] video analysis [17, 35], to name only a few.”
VRT: A Video Restoration Transformer (2022, methodology): “Some methods have tried to use Transformer for video modelling by extending the attention mechanism to the temporal dimension [2, 4, 45, 60, 68].”
Omnivore: A Single Model for Many Visual Modalities (2022, methodology): “The Transformer architecture [92] originally proposed for NLP tasks has been successfully applied in computer vision on images [11, 21, 70, 88, 93, 94], video [5, 8, 28, 29, 52, 66], and 3D data [60, 68, 103].”
Long-Short Temporal Contrastive Learning of Video Transformers (2021, background): “In the last few months, several adaptations of ViT to video have been proposed [3, 6, 44].”
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection (2021, background): “Vision transformers for video recognition have also recently shown strong results, but mostly [1, 3, 56, 59] rely on pre-training with large-scale external data (e.”
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, background): “In order to bypass the huge memory consumption brought by joint spatiotemporal attention, several pioneering work proposed efﬁcient alternatives, such as divided attention [4], axial attention [19, 4], factorized encoder [27, 2] and separable attention [54].”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, unknown): “…2021a;b; Liu et al., 2021a) have been popular for vision tasks and outperform many CNNs. Based on ViT, several prior works (Bertasius et al., 2021; Neimark et al., 2021; Sharir et al., 2021; Li et al., 2021b; Arnab et al., 2021; Bulat et al., 2021; Patrick et al., 2021; Zha et al., 2021) propose…”
SimVP: Simpler yet Better Video Prediction (2022, methodology): “VTN [42] applies sliding window attention on temporal dimension following a 2D spatial feature extractor.”
Aggregating Nested Transformers (2021, background): “ViT is given high expectation to improve vision tasks beyond image classiﬁcation, with existing studies on generative modeling [11, 27, 28, 39], video understanding [2, 38], segmentation and detection [30, 34, 53], interpretability [1, 6, 7].”
Latte: Latent Diffusion Transformer for Video Generation (2024, unknown): “In contrast to the temporal “inter-leaved fusion” design in Variant 1, this variant utilizes the “late fusion” approach to combine spatio-temporal information (Neimark et al, 2021; Simonyan and Zisserman, 2014).”