Co-training Transformer with Videos and Images Improves Action Recognition

By Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M. Dai, Ruoming Pang, Fei Sha. 2021

63 citations recorded by the source corpus.

In learning action recognition, models are typically pre-trained on object recognition with images, such as ImageNet, and later fine-tuned on target action recognition with videos. This approach has achieved good empirical performance especially with recent transformer-based video architectures. While recently many works aim to design more advanced transformer architectures for action recognition, less effort has been made on how to train video transformers. In this work, we explore several training paradigms and present two findings. First, video transformers benefit from joint training on diverse video datasets and label spaces (e.g., Kinetics is appearance-focused while SomethingSomething is motion-focused). Second, by further co-training with images (as single-frame videos), the video transformers learn even better video representations. We term this approach as Co-training Videos and Images for Action Recognition (CoVeR). In particular, when pretrained on ImageNet-21K based on the TimeSFormer architecture, CoVeR improves Kinetics-400 Top-1 Accuracy by 2.4%, Kinetics-600 by 2.3%, and SomethingSomething-v2 by 2.3%. When pretrained on larger-scale image datasets following previous state-of-the-art, CoVeR achieves best results on Kinetics-400 (87.2%), Kinetics-600 (87.9%), Kinetics-700 (79.8%), SomethingSomething-v2 (70.9%), and Moments-in-Time (46.1%), with a simple spatio-temporal video transformer.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Lu Yuan
2. Chen Sun
3. Yu-Gang Jiang
4. Haibin Ling
5. Yinpeng Chen
6. Cordelia Schmid
7. Anurag Arnab
8. Songyang Zhang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Cordelia Schmid
5. Lu Yuan
6. Xiaogang Wang
7. Yi Yang
8. Jianfeng Gao

In Multimodal Learning

1. Cordelia Schmid
2. Chen Sun
3. Anurag Arnab
4. Xiyang Dai
5. Lu Yuan
6. Jianlong Fu
7. Minghao Chen
8. Shiming Xiang

Citations in the corpus

• Multiview Transformers for Video Recognition (2022, background): “1%, which improves upon the previous highest result (CoVeR [82]) by 1.”
• BEVT: BERT Pretraining of Video Transformers (2021, background): “There are also a few very recent studies that extend image transformers for video understanding [2, 20, 37, 68].”
• Expanding Language-Image Pretrained Models for General Video Recognition (2022, methodology): “From the above fully-supervised experiments, we can observe that, our X-CLIP method achieves very competitive performance compared to prevailing video transformer models [56,55,54,49,21].”
• Masked Autoencoders As Spatiotemporal Learners (2022, unknown): “8 n/a n/a × 5 × 4 3000 supervised JFT-3B+SSv2+MiT+IN CoVeR [85] 16 × 448(2) 87.”