An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models

By Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, Yelong Shen. 2023

44 citations recorded by the source corpus.

Visual instruction tuning has recently shown encouraging progress with open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However, most existing studies of open-source LMM are performed using models with 13B parameters or smaller. In this paper we present an empirical study of scaling LLaVA up to 33B and 65B/70B, and share our findings from our explorations in image resolution, data mixing and parameter-efficient training methods such as LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language capabilities when completing real-world tasks in the wild. We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning. Additionally, the study highlights the importance of higher image resolutions and mixing multimodal-language data to improve LMM performance, and visual instruction tuning can sometimes improve LMM's pure language capability. We hope that this study makes state-of-the-art LMM research at a larger scale more accessible, thus helping establish stronger baselines for future research. Code and checkpoints will be made public.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ping Luo
2. Jianfeng Gao
3. Yu Qiao
4. Jifeng Dai
5. Chunyuan Li
6. Chunhua Shen
7. Yong Jae Lee
8. Luc Van Gool

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yu Qiao
3. Dacheng Tao
4. Xiaogang Wang
5. Trevor Darrell
6. Ming-Hsuan Yang
7. Ping Luo
8. Alan L. Yuille

In Large Language Models

1. Haotian Liu
2. Yong Jae Lee
3. Kurt Keutzer
4. Yu Qiao
5. Chuang Gan
6. Trevor Darrell
7. Chunyuan Li
8. Sheng Shen

In Multimodal Learning

1. Ping Luo
2. Chunyuan Li
3. Yu Qiao
4. Haotian Liu
5. Yong Jae Lee
6. Yuheng Li
7. Chuang Gan
8. Jifeng Dai

Citations in the corpus

• Improved Baselines with Visual Instruction Tuning (2023, methodology): “Recent works further demonstrate improved performance by scaling up the pretraining data [3, 13, 53], instruction-following data [13, 17, 28, 57], visual encoders [3], or language models [38], respectively.”
• Aligning Large Multimodal Models with Factually Augmented RLHF (2023, result): “…the absence of a negative feedback mechanism preventing the model from identifying hallucinations in its output, and the potential limitations of our Fact-RLHF reward model, especially when compared against the high-capacity oracle models in previous successful studies (Touvron et al., 2023b).”
• Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, unknown): “This method capitalizes on the foundational strengths of established models, significantly reducing the computational resources needed for initial training [160].”
• Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “Subsequently, GPT-4 [1], LLaVA series [68, 69, 76] and MiniGPT-4 [147] have brought in visual instruction tuning, to improve the instruction-following ability of VLLMs. Concurrently, models such as VisionLLM [113], KOSMOS-2 [87], and Qwen-VL et al .”