LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

By Yanwei Li, Chengyao Wang, Jiaya Jia. European Conference on Computer Vision, 2023

647 citations recorded by the source corpus.

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/486c2df78cbb770a90a55f7fa3fe19102fba2c24

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongsheng Li
2. Yu Liu
3. Guanglu Song
4. Yuhang Zang
5. Shengju Qian
6. Zhuofan Zong
7. Dahua Lin
8. Hao Shao

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Yu Liu
3. Guanglu Song
4. Xiaogang Wang
5. Dahua Lin
6. Dacheng Tao
7. Luc Van Gool
8. Yuhang Zang

Citations in the corpus

• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (2024, neutral): “Large Vision-Language Model. attempt to handle the long-context in LVLMs efficiently, like LLaMA-VID Li et al. (2023d), which utilizes cross-attention to effectively represent each video frame with two key tokens, the requirement for an additional fine-tuning stage obstructs its broad applicability…”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “While video LMMs excel in video understanding, they often do so at the expense of image performance [72, 76].”
• PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, background): “Commonly, they incorporate a projection network [30, 22, 19], inter-modality attention [17, 18] or a modality perceiver [45, 32, 12] as learnable interfaces.”
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, neutral): “While some recent works further sparsify vision tokens during the decoding (Ye et al., 2024; Chen et al., 2024b; Cai et al., 2024), they still ignore the guidance from the language tokens, which contradicts the multimodality paradigm.”
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “MLLMs have also been extended to various applications, including fine-grained localization [61, 24] such as object detection [72], video understanding [71, 29, 8], and image generation [20, 49].”
• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “The progress in M-LLMs for video data analysis, demonstrated by models such as Video-Chat [47], Video-ChatGPT [107], Video-LLaMA [48], LLaMA-VID [108], MotionGPT [109], LAVENDER [110], MovieChat [111], Vid2Seq [112], VideoLLM [113], and VTimeLLM [114], shows significant promise for the development…”
• ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, methodology): “LLaMA-VID [32] introduces the LLaVA-like three-stage training strategy and compresses each frame’s vision token into one to support long video understanding.”
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, neutral): “LLaMA-VID (Li et al., 2023d) employs an additional text decoder to embed the text query for cross-attention between frame features and compress the context token to one token per frame, while MovieChat (Song et al., 2023) and TimeChat (Ren et al., 2023b) develop memory modules and timestamp-aware…”
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, neutral): “Most existing works conduct supervised fine-tuning with only one type of data source, e.g., multi-image tuning of Man-tis [19] and multi-frame tuning of LLaMA-VID [31].”
• MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “Meanwhile, LLaMA-VID [22] presents the top performance for the generation tasks, whose G-Avg score reaches 4.22.”
• LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “Additionally, we assessed 13 models that natively support long videos: LLaMA-VID [19], MovieChat [34], LWM [20], Gemini 1.5 Pro [32], Kangaroo [21], Qwen2-VL-72B [37], Qwen2.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, neutral): “Specifically, we compare our VideoLLaMA 2 with VistaLLaMA (Ma et al., 2023), ChatUniVi (Jin et al., 2023b), LLaMA-VID (Li et al., 2023d), Video-LLaVA (Lin et al., 2023a), VideoChat2 (Li et al., 2023c), and LLaVA-NeXT-Video Se-4 ries 5 (Liu et al., 2024b).”
• Video-R1: Reinforcing Video Reasoning in MLLMs (2025, neutral): “For example, LLaMA-VID [15] proposes a dual-token strategy (context and content tokens) to compress video input representations, enabling vision-language models to efficiently handle long videos while retaining essential visual information.”