BEVT: BERT Pretraining of Video Transformers

By Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, Lu Yuan. 2021

269 citations recorded by the source corpus.

This paper studies the BERT pretraining of video transformers. It is a straightforward but worth-studying extension given the recent success from BERT pretraining of image transformers. We introduce BEVT which decouples video representation learning into spatial representation learning and temporal dynamics learning. In particular, BEVT first performs masked image modeling on image data, and then conducts masked image modeling jointly with masked video modeling on video data. This design is motivated by two observations: 1) transformers learned on image datasets provide decent spatial priors that can ease the learning of video transformers, which are often times computationally-intensive if trained from scratch; 2) discriminative clues, i.e., spatial and temporal information, needed to make correct predictions vary among different videos due to large intra-class and inter-class variations. We conduct extensive experiments on three challenging video benchmarks where BEVT achieves very promising results. On Kinetics 400, for which recognition mostly relies on discriminative spatial representations, BEVT achieves comparable results to strong supervised baselines. On Something-Something-V2 and Diving 48, which contain videos relying on temporal dynamics, BEVT outperforms by clear margins all alternative baselines and achieves state-of-the-art performance with a 71.4% and 87.2% Top-1 accuracy respectively. Code is available at https://github.com/xyzforever/BEVT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Visual Prompt Tuning (2022, background): “The triumph of the Transformer architecture also extends to various computer vision tasks, including image classification [15,39], object detection [8,36], semantic and panoptic segmentation [54,68,60], video understanding [20,61,17] and few-shot learning [14], surpassing previous state-of-the-art a”
Masked Autoencoders As Spatiotemporal Learners (2022, background): “In contrast, an extra tokenizer ( e.g ., dVAE [47, 9]), as is used in [3, 73], may require external data to train and additional domain knowledge to design ( e.g ., the dVAE used is a ConvNet [37]).”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “For spatiotemporal representation learning, VideoMAE [23] and BEVT [45] respectively extend MAE and BeiT to spatiotemporal space.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, methodology): “We use a simplified BEVT pre-training setup to test the effectiveness of our video tokens as targets for masked modeling.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “On a different note, Sermanet et al. [223] proposed a self-supervised technique for learning representations and robotic behaviors from unlabeled videos captured from various viewpoints. d) Spatial-temporal coherence of objects in videos: Wang et al. [224] introduced a self-supervised algorithm for…”
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers (2021, background): “Most recently, numerous works (Bao, Dong, and Wei 2021; He et al. 2021; Xie et al. 2021; Chen et al. 2022; Wang et al. 2022b; Dong et al. 2022; Baevski et al. 2022; Zheng et al. 2022) based on MIM have been concurrently developed, yet few studied the perceptual level of the prediction targets.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, methodology): “To learn spatial and temporal priors of videos in a decoupled way, BEVT [88] proposes a two-stage solu-tion that learns spatial representations with masked image modeling, then learns temporal representations with jointly masked image modeling and masked video modeling.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, background): “transformers are capable of capturing long-range dependencies among pixels/patches from input images effectively, which is arguably one of the main reasons that they outperform standard CNNs in vision tasks spanning from image classification [4, 12, 20, 22, 38, 49, 56] to object detection [3,5,43,44”