Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding

By Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, Li Yuan. Computer Vision and Pattern Recognition, 2023

464 citations recorded by the source corpus.

Large language models have demonstrated impressive universal capabilities across a wide range of open-ended tasks and have extended their utility to encompass multi-modal conversations. However, existing methods encounter challenges in effectively handling both image and video understanding, particularly with limited visual tokens. In this work, we introduce Chat-UniVi, a Unified Vision-language model capable of comprehending and engaging in conver-sations involving images and videos through a unified visual representation. Specifically, we employ a set of dynamic visual tokens to uniformly represent images and videos. This representation framework empowers the model to ef-ficiently utilize a limited number of visual tokens to simul-taneously capture the spatial details necessary for images and the comprehensive temporal relationship required for videos. Moreover, we leverage a multi-scale representation, enabling the model to perceive both high-level seman-tic concepts and low-level visual details. Notably, Chat-UniVi is trained on a mixed dataset containing both images and videos, allowing direct application to tasks involving both mediums without requiring any modifications. Exten-sive experimental results demonstrate that Chat- UniVi con-sistently outperforms even existing methods exclusively de-signed for either images or videos. Code is available at https://github.com/PKu-Yuan Group/Chat-UniVi.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/aad3d2e690f6c73f04a14622ceff51464bbc560e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhijian Liu
2. Hongxu Yin
3. Yukang Chen
4. Pavlo Molchanov
5. Jan Kautz
6. Haotian Tang
7. Zechun Liu
8. Ligeng Zhu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Zhijian Liu
3. Song Han
4. Dacheng Tao
5. Jan Kautz
6. Trevor Darrell
7. Yi Yang
8. Ming-Hsuan Yang

Citations in the corpus

• VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, unknown): “Compressive sparsity methods [10,12,14,30,42,44,50,53,59,65], attempt to compress the video into meaningful embeddings/representations with the minimum possible dimensionality.”
• TempCompass: Do Video LLMs Really Understand Videos? (2024, background): “Recent Video LLMs (Su et al., 2023; Li et al., 2023c; Zhang et al., 2023; Lin et al., 2023a; Li et al., 2023d; Maaz et al., 2023; Luo et al., 2023; Li et al., 2023g) primarily follow the End-to-End paradigm, with optional temporal modules to model the temporal information across frames.”
• LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, methodology): “, we prepared a long video and sampled a fixed number of frames.”
• TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, background): “With advancements in Large Language Models (LLMs), numerous studies have endeavored to integrate LLMs with a video encoder, thereby harnessing the powerful comprehension and generation capabilities of LLMs for video tasks [13, 18, 23, 28, 29, 40, 57].”
• PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, methodology): “Commonly, they incorporate a projection network [30, 22, 19], inter-modality attention [17, 18] or a modality perceiver [45, 32, 12] as learnable interfaces.”
• SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, unknown): “As shown in Table 7, SPHINX-Plus, despite being an image-based model, significantly outperforms existing models (Jin et al., 2023; Su et al., 2023; Zhang et al., 2023a) specifically tailored for video tasks.”
• Long Context Transfer from Language to Vision (2024, unknown): “1 2K Video-LLaMA [15] 32 32 LLaMA-2 4K Chat-UniVi [30] 112 64 Vicuna-1.”
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2023, background): “The presence of the glove and the ball on the bench indicates that theyare being used for a specific activityor exercise, and the dropping ofthe glovecould disruptthetrainingorpracticesession.”
• LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, unknown): “Some vision-language models enhance visual reasoning by optimizing the visual encoding strategy [23, 31, 34] to produce cognition-focused visual tokens.”
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “For the image-language pre-training stage, previous methods (Chen et al., 2023b; Peng et al., 2023; Wang et al., 2023; Chen et al., 2023a; Liu et al., 2024b; Dong et al., 2024) usually use two steps for alignment and finetuning.”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, unknown): “Specifically, we compare our VideoLLaMA 2 with VistaLLaMA (Ma et al., 2023), ChatUniVi (Jin et al., 2023b), LLaMA-VID (Li et al., 2023d), Video-LLaVA (Lin et al., 2023a), VideoChat2 (Li et al., 2023c), and LLaVA-NeXT-Video Se-4 ries 5 (Liu et al., 2024b).”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, unknown): “5 [23], LLaVA-NeXT-Video [73], VITA-1.”