Visual Instruction Tuning
By Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee. Neural Information Processing Systems, 2023
10,396 citations recorded by the source corpus.
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
View Paper: https://arxiv.org/abs/2304.08485
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Sergey Levine
- Yoshua Bengio
- Quoc V. Le
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Alan L. Yuille
In Multimodal Learning
Citations in the corpus
- The False Promise of Imitating Proprietary LLMs (2023, methodology): “For example, the training process of many recent vision-language model (Li et al., 2022; Liu et al., 2023; Ye et al., 2023; Zhu et al., 2023) includes ChatGPT or GPT-4 outputs at some stages.”
- Improved Baselines with Visual Instruction Tuning (2023, positive): “∼ 600K [35] or ∼ 6M [60]), while some recent works pretrain the vision-language connector for a specific language model on a large amount of image-text pairs ( e.g .”
- Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “Meanwhile, some work [57, 58, 59] introduce visual instruction tuning to enhance instruction following capabilities.”
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “The recent open-sourcing of LLMs [70–73, 100] further accelerates the emergence of public MLLMs [ 21, 47, 104].”
- LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “Aided by our joint training strategy, LLaMAAdapter V2 requires no high-quality multi-modal instruction data like MiniGPT-4 [78] and LLaVA [38], but only image-text pairs and instruction-following data, as compared in Tab.”
- Are aligned neural networks adversarially aligned? (2023, methodology): “These visual tokens may be passed directly as an input to the model [Zhu et al., 2023, Liu et al., 2023], surrounded by special templates (e.g., “ <img> . . . <\img> ”) to delineate their modality, or combined internal to the model via learned adaptation prompts [Gao et al., 2023].”
- PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, methodology): “The daunting challenge in the development of an end-to-end multi-modal LLM is procuring large-scale multi-modal instruction-following data, vital for representation learning, aligning latent spaces, and orienting the model to adhere to human intentions [1,7,25,27,56].”
- Depth Anything V2 (2024, neutral): “Lastly, we believe it is also a potential testbed for the 3D awareness of future multimodal LLMs [41, 21, 3].”
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, methodology): “We evaluate the general performance of MMICL on both MME (Fu et al., 2023) and MMBench (Liu et al., 2023d) benchmarks 2 .”
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “Propelled by the principles of scaling laws in neural language models [30], several Large Language Models [9, 76, 70, 27, 63, 64] have been proposed, embodying the principle that increasing the scale of models tends to yield enhanced performance outcomes.”
- 3D-LLM: Injecting the 3D World into Large Language Models (2023, background): “Our work is also related to vision-language pre-trained models that connect images and natural language [30, 31, 18, 36, 25].”
- DriveLM: Driving with Graph Visual Question Answering (2023, methodology): “DriveLM-Agent employs a trajectory tokenizer that can be applied to any general VLM [40, 45, 53, 83], coupled with a graph prompting scheme that models logical dependencies as context inputs for VLMs.”
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (2023, methodology): “To train the hallucination revisor, we randomly selected 5000 image-text pairs from LLaVA-150k (Liu et al., 2023c), ensuring that these images were different from the ones used in testing.”
- Multimodal Large Language Models: A Survey (2023, background): “LLaVA [46] and MiniGPT-4 are similar, as both aim to achieve multimodal instruction fine-tuning.”
- SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models (2023, methodology): “3) Employing advanced image encoders for extracting visual embeddings, e.g., the CLIP encoder (Radford et al., 2021) in LLaVA (Liu et al., 2023c) and the Q-Former (Li et al., 2023d) in MiniGPT-4 (Zhu et al., 2023).”
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “There are three primary approaches to connector design: Resamplers [5], Q-Formers [10, 31], and MLP Projectors [73, 75].”