InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

By Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu and 3 more. International Conference on Learning Representations, 2023

513 citations recorded by the source corpus.

This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. The InternVid dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is to develop a scalable approach to autonomously build a high-quality video-text dataset with large language models (LLM), thereby showcasing its efficacy in learning video-language representation at scale. Specifically, we utilize a multi-scale approach to generate video-related descriptions. Furthermore, we introduce ViCLIP, a video-text representation learning model based on ViT-L. Learned on InternVid via contrastive learning, this model demonstrates leading zero-shot action recognition and competitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video generation research. These proposed resources provide a tool for researchers and practitioners interested in multimodal video understanding and generation.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/369b449415d50387fba048bbd4d26ee890df84b5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, background): “…are a few open-source video datasets collected from the Internet for video understanding, such as HowTo100M [33], HD-VILA-100M [56], and InterVid [51], there exist many issues for video generation, e.g., poor picture quality and caption, multiple clips in one video, and static frames or slides.”
Video Instruction Tuning With Synthetic Data (2024, unknown): “Specifically, these ten sources are HD-VILA-100M (Xue et al., 2022), InternVid-10M (Wang et al., 2023), VidOR (Shang et al., 2019), VIDAL (YouTube Shorts)(Zhu et al., 2023a), YouCook2(Zhou & Corso, 2017), Charades (Sigurdsson et al., 2016), ActivityNet (Caba Heilbron et al., 2015), Kinetics-700…”
VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “We use ViCLIP [105] to calculate the video feature and the temporal style description feature similarity to reflect temporal style consistency.”
LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, unknown): “Specifically, these ten sources are HD-VILA-100M (Xue et al., 2022), InternVid-10M (Wang et al., 2023), VidOR (Shang et al., 2019), VIDAL (YouTube Shorts)(Zhu et al., 2023a), YouCook2(Zhou & Corso, 2017), Charades (Sigurdsson et al., 2016), ActivityNet (Caba Heilbron et al., 2015), Kinetics-700…”
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers (2024, methodology): “With the aforementioned development on vision-language models, some concurrent works [7, 72, 75] also leverage these models for video captioning.”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “In addition to the aforementioned training data in Stage1, we further introduce 2M image captions (COCO [46], Visual Genome [34], and SBU [60]) and 10M video captions (InternVid [78]), to enrich the caption diversity.”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “Following previous meth-ods [89, 99, 117], we report the top-1 accuracy and the mean of top-1 and top-5 accuracy on Kinetics-400/600/700 [15– 17].”
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “InternVid [18] proposes a scalable approach for autonomously constructing a video-text dataset using large language models, yielding 234 million video clips accompanied by text descriptions for basic video understanding tasks such as recognition and retrieval, as well as T2V generation.”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Most video generation works use diffusion-based methods for text-to-video [4, 5, 25, 30, 33, 56, 67, 69, 70, 80, 81, 81, 85] and video-to-video editing [15, 23, 24, 39].”
VTimeLLM: Empower LLM to Grasp Video Moments (2023, background): “Recently, we identified the InternVid-10M-FLT [27] dataset, which offers a viable solution for our boundary-aware training.”
VideoMamba: State Space Model for Efficient Video Understanding (2024, unknown): “Beyond action recognition, large-scale video-text datasets [10, 13, 58, 84, 87, 89] extend video understanding into multi-modality tasks like video captioning, retrieval, and question answering.”
Emu3: Next-Token Prediction is All You Need (2024, methodology): “The tokenizer is trained end-to-end on the LAION-High-Resolution 6 image dataset and the InternVid [89] video dataset using combined objective functions of L2 loss, LPIPS perceptual loss [104], GAN loss, and commitment loss [23].”