LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

By Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, Chunyuan Li. arXiv.org, 2024

619 citations recorded by the source corpus.

Visual instruction tuning has made considerable strides in enhancing the capabilities of Large Multimodal Models (LMMs). However, existing open LMMs largely focus on single-image tasks, their applications to multi-image scenarios remains less explored. Additionally, prior LMM research separately tackles different scenarios, leaving it impossible to generalize cross scenarios with new emerging capabilities. To this end, we introduce LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame (video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To enable these capabilities, we regard the interleaved data format as a general template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4 primary domains with 14 tasks and 41 datasets. We also curate the LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading results in multi-image, video, and 3D benchmarks, while maintaining the performance of single-image tasks. Besides, our model also exhibits several emerging capabilities, e.g., transferring tasks across different settings and modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/2e230519ef242e592b3c19765639008c7046424f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Peng Gao
2. Renrui Zhang
3. Pan Lu
4. Ziyu Guo
5. Sicong Leng
6. Wenqi Zhang
7. Li Bing
8. Yichi Zhang

Citations in the corpus

• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “Previous studies [28, 29, 93, 166] have demonstrated that general MLLMs with robust image understanding capabilities can achieve remarkable performance on video understanding tasks, even with minimal or no dedicated video training data.”
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “…removing the redundant texts and input diagrams on 120 randomly selected problems, for closed-sourced [47, 22, 3] and open-sourced [21, 38, 16] MLLMs. a fast-evolving track, exemplified by the latest GPT-4V [47], Gemini [22], and the open-source LLaVA [39, 34, 32, 30] and SPHINX [36, 21].”
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, neutral): “…capabilities (Liu et al., 2023c,b, 2024a; Li et al., 2024b; Dai et al., 2023; Zhu et al., 2023; Zhang et al., 2023; Gao et al., 2023; Ye et al., 2023a,b; Zhao et al., 2023; Li et al., 2023; Monajatipoor et al., 2023; Zhao et al., 2024; Li et al., 2024c; Lin et al., 2024; Zhang et al., 2024).”