InternVideo: General Video Foundation Models via Generative and Discriminative Learning

By Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang and 7 more. arXiv.org, 2022

545 citations recorded by the source corpus.

The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, which are limited for dynamic and complex video-level understanding tasks. To fill the gap, we present general video foundation models, InternVideo, by taking advantage of both generative and discriminative self-supervised video learning. Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications. Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications. Especially, our methods can obtain 91.1% and 77.2% top-1 accuracy on the challenging Kinetics-400 and Something-Something V2 benchmarks, respectively. All of these results effectively show the generality of our InternVideo for video understanding. The code will be released at https://github.com/OpenGVLab/InternVideo .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/325d8e9501af05e594bd668b6cd6d43ed42c8b4d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, methodology): “Conversely, InternVideo [30] and UMT [19] combined masked modeling with crossmodal contrastive learning, leading to competitve performance in both video-only and video-language tasks.”
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “The common paradigm in the field of video foundation models is now characterized by the combination of extensive large-scale video-language pretraining, followed by fine-tuning on specific downstream tasks [45,46,48,57,77,89,96,111].”
VideoMamba: State Space Model for Efficient Video Understanding (2024, unknown): “Recent models [42,44,52,85] have focused on e ﬃ cient video transformers, with innovations like VideoSwin’s window attention [52] and Uni-Former’s integration of convolution and self-attention [44], balancing computational e ﬃ ciency with performance.”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, methodology): “In zero-shot text to video retrieval, LanguageBind achieves superior performance on four datasets, surpassing InterVideo (Wang et al., 2022c) by 1.9% on MSR-VTT (Xu et al., 2016), 8.8% on MSVD (Chen & Dolan, 2011), 6.3% on DiDeMo (Anne Hendricks et al., 2017), and 4.4% on ActivityNet (Caba Heil-bron”
VideoChat: chat-centric video understanding (2023, unknown): “We employ several vision models to convert video data into textual format.”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, unknown): “Our results indicate that VideoChat2 holds its own against current SOTA methods [77, 94, 96] on these complex reasoning tasks, which underscores the effectiveness and robustness of VideoChat2, especially for long videos.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “InternVideo [49] proposes training video-language models jointly with masked video modeling and contrastive learning objectives.”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “Besides, pretraining UMT [39] with our Panda-5M also outperforms the existing state-of-the-art methods [49, 50, 74] which are pretrained with much more vision-text data pairs ( i.e ., > 100M).”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2023, methodology): “Additionally, they construct a video-specific dataset using off-the-shelf vision-language models [27, 4, 28, 26] for generating noisy detailed textual descriptions to enhance the training of video-centric conversational models.”