LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

By Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, Tongfei Sun. 2023

313 citations recorded by the source corpus.

Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, each containing question-answer pairs for text-rich images. By combining our collected data with previous multi-modal instruction-following data, our model, LLaVAR, substantially improves the LLaVA model's capability on text-based VQA datasets (up to 20% accuracy improvement) while achieving an accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following evaluation also demonstrates the improvement of our model on both natural images and text-rich images. Through qualitative analysis, LLaVAR shows promising interaction (e.g., reasoning, writing, and elaboration) skills with humans based on the latest real-world online content that combines text and images. We make our code/data/models publicly available at https://llavar.github.io/.

Subfields: Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Multimodal Learning

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, methodology): “LLaVA’s pipeline has been employed to expand to textual understanding [56], million-scales [57], and region-level conversations [8].”
MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, unknown): “LLaVA-Med (Li et al., 2023d), LLaVAR (Zhang et al., 2023h), mPLUG-DocOwl( I D ) (Ye et al., 2023a), DLP (Jian et al., 2023), ChatSpot (Zhao et al., 2023b), OpenFlamingo (Awadalla et al., 2023), Chinese-LLaVA (LinkSoul-AI., 2023), ASM (Wang et al., 2023c), BLIVA (hu2, 2023), IDEFICS (IDEFICS, 2023), ”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…COCO-Caption [214], OpenImages-Caption [116], Objects365-Caption [208], TextCap [211], Laion-ZH [203], Laion-EN [203], Laion-COCO [204], LLaVAR [305], InternVL-SA-1B-Caption [113], Captioning MMInstruct [155], GRIT-Caption [194], ShareGPT4V [29], LVIS-Instruct-4V [244], ShareCaptioner [29],…”
Knowledge Graph Prompting for Multi-Document Question Answering (2023, methodology): “As the predicted answer may not overlap with the ground-truth one, we additionally check the correctness of the answer following (Liu et al. 2023b; Dubois et al. 2023; Zhang et al. 2023) by prompting the LLM.”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…to judge LMMs on skills that require external knowledge (Schwenk et al., 2022; Shah et al., 2019), common sense reasoning (Zellers et al., 2019; Yin et al., 2021), scientific-knowledge (Lu et al., 2022; Kembhavi et al., 2017, 2016), medical understanding (Zhang et al., 2023b; Lau et al., 2018).”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “Following Liu et al. [36], Zhang et al. [72], we em-16”
OCRBench: on the hidden mystery of OCR in large multimodal models (2023, unknown): “Additionally, [22, 23] emphasize modality collaboration for image and text.”
Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, background): “Most recently, researchers have been creating high-quality, diverse multi-modal datasets from GPT4 and GPT-4V [48] to fine-tune open-source LVLMs, including LLaVA [33], MiniGPT4 [63], Mplug-Owl [50], LRV-Instruction [29], LLaVAR [60] and other works [12, 25, 37, 52].”
Generative Multimodal Models are In-Context Learners (2023, background): “( b ) Multimodal chat data: GPT-assisted visual instruction [49, 93], language instruction [4, 72], clock reading [83], and video chat [44].”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, unknown): “Concurrently, to maintain OpenFlamingo’s in-context learning capacity, we retrieve in-context examples for each triplet, which often lack correlated context, such as a visual question-answer data sample in VQAv2 [3].”
Yi: Open Foundation Models by 01.AI (2024, background): “F e a t u r e s million image-text pairsn from diverse sources, e.g. , CLLaVA [45], LLaVAR [91], Flickr [85], VQAv2 [25], RefCOCO [37], Visual7w [95] and so on.”