VidTr: Video Transformer Without Convolutions

By Xinyu Li, Yanyi Zhang, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, Joseph Tighe. 2021

226 citations recorded by the source corpus.

We introduce Video Transformer (VidTr) with separable-attention for video classification. Comparing with commonly used 3D networks, VidTr is able to aggregate spatiotemporal information via stacked attentions and provide better performance with higher efficiency. We first introduce the vanilla video transformer and show that transformer module is able to perform spatio-temporal modeling from raw pixels, but with heavy memory usage. We then present VidTr which reduces the memory cost by 3.3× while keeping the same performance. To further optimize the model, we propose the standard deviation based topK pooling for attention (pooltopK_std), which reduces the computation by dropping non-informative features along temporal dimension. VidTr achieves state-of-the-art performance on five commonly used datasets with lower computational requirement, showing both the efficiency and effectiveness of our design. Finally, error analysis and visualization show that VidTr is especially good at predicting actions that require long-term temporal reasoning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Multiview Transformers for Video Recognition (2022, neutral): “…[52] 79.4 94.4 – – SlowFast R101-NL [23] 79.8 93.9 10 × 3 7.02 X3D-XXL [22] 80.4 94.6 SlowFast R50 [23,77] 61.7 – TimeSformer-HR [6] 62.5 – VidTR [83] 63.0 – ViViT-L FE [3] 65.9 89.9 MViT [21] 67.7 90.9 MFormer-L [51] 68 increasing t for all other views) will roughly reduce the input tokens by…”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, background): “[4, 6, 87] make attempts to apply Transformers on video recognition in a convolution-free fashion.”
Video Swin Transformer (2021, background): “The great success of image Transformers has led to investigation of Transformer-based architectures for videobased recognition tasks [1, 3, 9, 25, 29].”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, methodology): “Recent methods propose well-designed modules to enhance the temporal modeling ability for 2D CNNs (Wang et al., 2016; Lin et al., 2019; Luo & Yuille, 2019; Jiang et al., 2019; Liu et al., 2020a; Li et al., 2020b; Kwon et al., 2020; Li et al., 2020a; 2021a; Wang et al., 2020b).”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “Researchers thus have made plenty of efforts to decrease computation complexity [153], [156], [161], [164], [168], [176] and memory cost [154], [161], [164], [165], [176].”
Blending Anti-Aliasing into Vision Transformer (2021, methodology): “Given its superior performance on classiﬁcation, recent works apply transformer architecture to various vision tasks, including object detection [16, 17], segmentation [3, 18, 19], point cloud processing [20], image generation [21–24] and so on.”
Anticipative Video Transformer (2021, background): “In contrast, AVT is an end-to-end transformer architecture for video—to our knowledge the first (concurrent with [4, 7, 19, 52, 63]).”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, background): “While obtaining superior performance, the computational cost of vision transformers is still intensive and scales up quickly as the numbers of patches, self-attention heads and transformer blocks increase.”