Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers

By Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang and 1 more. Computer Vision and Pattern Recognition, 2024

424 citations recorded by the source corpus.

The quality of the data and annotation upper-bounds the quality of a downstream model. While there exist large text corpora and image-text pairs, high-quality video-text data is much harder to collect. First of all, manual labeling is more time-consuming, as it requires an annotator to watch an entire video. Second, videos have a temporal dimension, consisting of several scenes stacked together, and showing multiple actions. Accordingly, to establish a video dataset with high-quality captions, we propose an automatic approach leveraging multimodal inputs, such as textual video description, subtitles, and individual video frames. Specifically, we curate 3.8M high-resolution videos from the publicly available HD-VILA-100M dataset. We then split them into semantically consistent video clips, and apply multiple cross-modality teacher models to obtain captions for each video. Next, we finetune a retrieval model on a small subset where the best caption of each video is manually selected and then employ the model in the whole dataset to select the best caption as the annotation. In this way, we get 70M videos paired with high-quality text captions. We dub the dataset as Panda-70M. We show the value of the proposed dataset on three downstream tasks: video captioning, video and text retrieval, and text-driven video generation. The models trained on the proposed data score substantially better on the majority of metrics across all the tasks.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8eae862d9669e7001eeee17b49fba793df9672c4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Ming Ding
2. Jie Tang
3. Haodong Duan
4. Yuhang Zang
5. Wenyi Hong
6. Wendi Zheng
7. Xiao-wen Dong
8. Jiaqi Wang

Citations in the corpus

• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, methodology): “First, we use the Panda70M video captioning model (Chen et al., 2024b) to generate short captions for the videos.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “The video-text sources include Panda-70M (Chen et al., 2024b), VIDAL-10M (Zhu et al., 2023a) WebVid-10M (Bain et al., 2021), and InternVid-10M (Wang et al., 2023), while the image-text sources include CC-3M (Changpinyo et al., 2021) and DCI (Urbanek et al., 2023).”
• ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, methodology): “For fairness and simplicity, we integrate 28K high-quality video-caption data related to complex scenes (Panda-70M [12], Ego4D [18], and BDD100K [66]) of ShareGPT4Video to replace the captions data in the VideoChatGPT-100K [42] conversation data with an equivalent number.”
• Video Instruction Tuning With Synthetic Data (2024, neutral): “More recent research (Chen et al., 2024b) has prompted video-language models such as BLIP-2 (Li et al., 2023), VideoChat (Li et al., 2024e), Video-LLaMA (Zhang et al., 2023), and MiniGPT-4 (Zhu et al., 2023b) to generate video captions.”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “Specifically, following the pipeline proposed in [24], we first filter 68K dynamic videos from Panda-70M [112] dataset by optical flow, and then employ Qwen2-VL-72B [30] to generate diverse dense captions and QAs for each video from the aspects of temporal understanding, spatial understanding,…”
• LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, neutral): “More recent research (Chen et al., 2024b) has prompted video-language models such as BLIP-2 (Li et al., 2023), VideoChat (Li et al., 2024e), Video-LLaMA (Zhang et al., 2023), and MiniGPT-4 (Zhu et al., 2023b) to generate video captions.”
• OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, background): “Meanwhile, Panda-70M [10] contains lots of low aesthetics, static, flickering, excessively dynamic or low clarity videos along with brief captions , while our OpenVid - 1M features high-quality videos coupled with expressive captions, achieved through the expansive multimodal model LLaVA-v1.”
• HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “Most previous work focus on providing either brief captions [14, 50] or dense captions [93, 9, 10].”