Long Context Transfer from Language to Vision

By Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, Ziwei Liu. Trans. Mach. Learn. Res., 2024

531 citations recorded by the source corpus.

Video sequences offer valuable temporal information, but existing large multimodal models (LMMs) fall short in understanding extremely long videos. Many works address this by reducing the number of visual tokens using visual resamplers. Alternatively, in this paper, we approach this problem from the perspective of the language model. By simply extrapolating the context length of the language backbone, we enable LMMs to comprehend orders of magnitude more visual tokens without any video training. We call this phenomenon long context transfer and carefully ablate its properties. To effectively measure LMMs' ability to generalize to long contexts in the vision modality, we develop V-NIAH (Visual Needle-In-A-Haystack), a purely synthetic long vision benchmark inspired by the language model's NIAH test. Our proposed Long Video Assistant (LongVA) can process 2000 frames or over 200K visual tokens without additional complexities. With its extended context length, LongVA achieves state-of-the-art performance on Video-MME among 7B-scale models by densely sampling more input frames. Our work is open-sourced at https://github.com/EvolvingLMMs-Lab/LongVA.

View Paper: https://www.semanticscholar.org/paper/d081584960c42f7793502bb496e46f682e3e43b3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, unknown): “For open-source models, we evaluate models from InternVL2 [14], ViLA [45], LongViLA [91], LongVA [101], LLaVA-OneVision [40], and LLaVA-Video [104].”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, background): “For instance, LongVA (Zhang et al., 2024a) depends on long-context LLMs and trains models on short-context data.”
Video-R1: Reinforcing Video Reasoning in MLLMs (2025, unknown): “LongVA [28] extends the context window of language backbones to process significantly longer video sequences without specialized video training, offering a language-centric solution to long-range temporal reasoning.”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “For example, some [13] directly extend the context window of LLMs, while others [24, 142–149] introduce video token compression techniques that perform pooling across spatial, temporal, or both dimensions.”