PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning
By Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See-Kiong Ng, Jiashi Feng. arXiv.org, 2024
360 citations recorded by the source corpus.
Vision-language pre-training has significantly elevated performance across a wide range of image-language applications. Yet, the pre-training process for video-related tasks demands exceptionally large computational and data resources, which hinders the progress of video-language models. This paper investigates a straight-forward, highly efficient, and resource-light approach to adapting an existing image-language pre-trained model for dense video understanding. Our preliminary experiments reveal that directly fine-tuning pre-trained image-language models with multiple frames as inputs on video datasets leads to performance saturation or even a drop. Our further investigation reveals that it is largely attributed to the bias of learned high-norm visual features. Motivated by this finding, we propose a simple but effective pooling strategy to smooth the feature distribution along the temporal dimension and thus reduce the dominant impacts from the extreme features. The new model is termed Pooling LLaVA, or PLLaVA in short. PLLaVA achieves new state-of-the-art performance on modern benchmark datasets for both video question-answer and captioning tasks. Notably, on the recent popular VideoChatGPT benchmark, PLLaVA achieves a score of 3.48 out of 5 on average of five evaluated dimensions, exceeding the previous SOTA results from GPT4V (IG-VLM) by 9%. On the latest multi-choice benchmark MVBench, PLLaVA achieves 58.1% accuracy on average across 20 sub-tasks, 14.5% higher than GPT4V (IG-VLM). Code is available at https://pllava.github.io/
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/9d29da83aba362c728c36f4dea9dde678ae3e2b2
Citations in the corpus
- mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, neutral): “…format. we include VideoChat2 (Li et al., 2023c), Video-LLaMA2 (Cheng et al., 2024), Video-ChatGPT (Maaz et al., 2023), ShareGPT4Video (Chen et al., 2024c), PLLaVA (Xu et al., 2024), Idefics2 (Laurenccon et al., 2024), Mantis-SigLIP (Jiang et al., 2024) and LLAVA-Interleave (Li et al., 2024a).”
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “Our extensive evaluation under-scores the need for further advancements in handling longer multimodal data and we hope Video-MME will inspire future research and development in improving the capabilities of MLLMs.”
- LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, neutral): “The LLaVA-NeXT-Video (Zhang et al., 2024e) and PLLaVA (Xu et al., 2024a) models address this by simly considering average pooling to reduce the number of tokens representing each frame.”
- LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “…that contains human-annotated action-related QA pairs derived from ActivityNet dataset, EgoSchema [98] and MLVU [169] focusing on long video understanding, PerceptionTest [115] designed to evaluate the perception skills, VideoMME [29] and NeXTQA [141] containing diverse video domains and…”
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, neutral): “The scene shifts to a showroom setting, showcasing two blue cars, followed by a man in a black coat discussing features in front of a laptop.”
- LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “The annotation of long videos presents unique challenges compared to short-form content, requiring careful consideration of temporal dynamics and comprehensive coverage.”