Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

By Hang Zhang, Xin Li, Lidong Bing. Conference on Empirical Methods in Natural Language Processing, 2023

1,876 citations recorded by the source corpus.

We present Video-LLaMA a multi-modal framework that empowers Large Language Models (LLMs) with the capability of understanding both visual and auditory content in the video. Video-LLaMA bootstraps cross-modal training from the frozen pre-trained visual and audio encoders and the frozen LLMs. Unlike previous works that complement LLMs to process the visual or audio signals only, Video-LLaMA enables video comprehension by tackling two challenges: (1) capturing the temporal changes in visual scenes, (2) integrating audio-visual signals. To counter the first challenge, we propose a Video Q-former to assemble a pre-trained image encoder into our video encoder and introduce a video-to-text generation task to learn video-language correspondence. For the second challenge, we leverage ImageBind, a universal embedding model aligning multiple modalities, as the pre-trained audio encoder and introduce an Audio Q-former on top of ImageBind to learn reasonable auditory query embeddings for the LLM module. To align the output of both visual and audio encoders with LLM's embedding space, we first train Video-LLaMA on massive video/image-caption pairs and then tune our model with visual-instruction datasets of moderate amount but higher quality. We found Video-LLaMA shows the ability to perceive and comprehend video content and generate meaningful responses grounded in the visual and auditory information presented in the videos.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5d321194696f1f75cf9da045e6022b2f20ba5b9c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “MLLMs have also been extended to various applications, including fine-grained localization [61, 24] such as object detection [72], video understanding [71, 29, 8], and image generation [20, 49].”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Additionally, they can adapt to changing road conditions with a level of understanding that closely resembles human intuition.”
LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples (2023, background): “Moreover, this also formulate a new paradigm in the community that large pre-trained generative models contain rich knowledge to adaptive many task even some different modalities [23].”
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, methodology): “We evaluate all image LMMs [Ye et al., 2023, Dai et al., 2023, Li et al., 2023b, Liu et al., 2023a,b, 2024b] with 8 input frames.”
LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, neutral): “Compared to the construction of image-language instruction-following data, obtaining high-quality video-language instruction-following data is challenging (Zhang et al., 2023; Li et al., 2024e).”
NExT-GPT: Any-to-Any Multimodal LLM (2023, methodology): “, Video-Chat [44] and Video-LLaMA [104]), audios (e.”
SVIT: Scaling up Visual Instruction Tuning (2023, background): “There are also some works that contribute multimodal instruction data of videos[14], RGB-D images[14], speech[28], audio[29], et al.”
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, methodology): “In addition, we calculate the optical flow scores and image aesthetic scores of all training videos and dynamically adjust the threshold ranges during training to ensure the fluency and aesthetic quality of the generated videos.”
A Comprehensive Overview of Large Language Models (2023, neutral): “Inspired by the success of LLMs in natural language processing applications, an increasing number of research works are now facilitating LLMs to perceive different modalities of information like image [200], [201], [202], video [203], [204], [205], audio [206], [205], [207], etc.”
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, methodology): “Firstly , their rigid compression converting video tokens to a fixed number (e.g. 32 [57]) is unsuitable for long-form video input [37].”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (2023, methodology): “To support video understanding in LLMs, Video-LLaMA [62] and VideoChat [30] attempt to utilize BLIP-2 for video embedding extraction, while Video-ChatGPT [39] proposes spatial and temporal pooling for video features.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Similarly, models such as Video-Chat [47] and Video-LLaMA [48] are designed for video comprehension, whereas SpeechGPT [49] is tailored for audio processing.”
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, background): “Commonly, they incorporate a projection network [30, 22, 19], inter-modality attention [17, 18] or a modality perceiver [45, 32, 12] as learnable interfaces.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “This includes video inputs [110, 126], audio [80] or both [216].”