Multiview Transformers for Video Recognition

By Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, Cordelia Schmid. 2022

292 citations recorded by the source corpus.

Video understanding requires reasoning at multiple spatiotemporal resolutions – from short fine-grained motions to events taking place over longer durations. Although transformer architectures have recently advanced the state-of-the-art, they have not explicitly modelled different spatiotemporal resolutions. To this end, we present Multiview Transformers for Video Recognition (MTV). Our model consists of separate encoders to represent different views of the input video with lateral connections to fuse information across views. We present thorough ablation studies of our model and show that MTV consistently performs better than single-view counterparts in terms of accuracy and computational cost across a range of model sizes. Furthermore, we achieve state-of-the-art results on six standard datasets, and improve even further with large-scale pretraining. Code and checkpoints are available at: https://github.com/google-research/scenic.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[134] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “[2] which learns to represent audio independently from vision (and so cannot early-fuse them), along with the larger MTV-Huge model [122] by 1.”
Masked Autoencoders As Spatiotemporal Learners (2022, unknown): “2 n/a n/a × 3 × 1 n/a supervised WTS-60M MTV-H [82] 32 × 280(2) 89.”
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, methodology): “Beyond minimal adaptation from ViT, some recent work introduced more prior to the design of space-time transformers, including trajectory [29], multi-scale [23, 11] and multi-view [49].”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, background): “image or action classifiers [51, 20], object detectors [14], or object trackers [46], opening up the path towards general perception and reasoning models.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, background): “Compared with previous related work [6, 8, 9], it greatly lifts the generality of video foundation models to a new level, by achieving state-of-the-art performance on nearly 40 datasets covering 10 different tasks.”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, unknown): “5 MTV-B(320p)[80] IN21K+K400 32×3×4 11160 68.”
SCENIC: A JAX Library for Computer Vision Research and Beyond (2021, methodology): “On top of that, SCENIC has been used in numerous Google projects and research papers such as ViViT [6], OmniNet [28], TokenLearner [25], MBT [21], PolyViT [19], MTV [31], studies on scaling behaviour [5, 10, 29] and efficiency [9] of various models among others [15–18, 20].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “[169] introduced Multiview Transformers, consisting of multiple individual”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, methodology): “3% top-1 accuracy while using 5 × fewer FLOPs compared to the current state-of-the-art method MTV-H [54].”
Multimodal Learning With Transformers: A Survey (2022, background): “[288] present an efficient solution that sequentially fuses information between all pairs of two adjacent views in ascending order of sequence”
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking (2023, methodology): “On the Kinetics datasets, our VideoMAE V2 achieves the best performance among these methods without using extra inhouse labeled data, and quite competitive performance to the leading performance MTV [82] trained with extra 60M labeled videos.”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, methodology): “By combining some successful designs in CNN architectures and the new self-attention mechanism, vision Transformers have shown leading performance on various vision tasks such as image classification [12, 40, 46], object detection [66, 39], semantic segmentation [6, 8] and video understanding [60, 1”