MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

By Zhiyang Xu, Ying Shen, Lifu Huang. 2022

142 citations recorded by the source corpus.

Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multimodal tasks. In this work, we introduce MultiInstruct, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expert-written instructions. We take OFA as the base pre-trained model for multimodal instruction tuning, and to further improve its zero-shot performance, we explore multiple transfer learning strategies to leverage the large-scale Natural Instructions dataset. Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset. We also design a new evaluation metric – Sensitivity, to evaluate how sensitive the model is to the variety of instructions. Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “…inputs (Hao et al., 2022; Huang et al., 2023; Chen et al., 2023b) with parameter-efficient tuning (Hu et al., 2022; Alayrac et al., 2022; Li et al., 2023d; Zhang et al., 2023e; Zhu et al., 2023a; Ye et al., 2023) and instruction tuning (Xu et al., 2023b; Liu et al., 2023c; Dai et al., 2023a).”
MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023, methodology): “The notion of instruction tuning in multi-modal models was initially introduced in the work called Multi-Instruct [50], which encompassed a wide range of multi-modal tasks [18, 56, 41, 27, 12] involving visual understanding and multi-modal reasoning, such as Visual Question Answering [18, 56, 23].”
A Survey on In-context Learning (2022, background): “Recent explorations first generate instruction tuning datasets transforming existing vision-language task dataset (Xu et al., 2022; Li et al., 2023a) or with power LLMs such as GPT-4 (Liu et al.”
MultiModal-GPT: A Vision and Language Model for Dialogue with Humans (2023, background): “Other vision and language instruction datasets, such as MultiInstruct [17] can also be explored, and we left it as future work.”
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (2023, background): “Multiinstruct (Xu et al., 2023) introduces instruction tuning to enhance the performance of VLMs in instruction-following ability.”
A Comprehensive Overview of Large Language Models (2023, unknown): “LongLoRA [174] proposes shift-short attention, used during fine-tuning to reduce dense attention costs, while the model during inference can use dense attention and achieve similar performance as full attention fine-tuning.”
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023, background): “A separate work is MultiInstruct [51], which performs vision-language instruction tuning without a pretrained LLM, leading to less competitive performance.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, unknown): “The concept of instruction tuning in multi-modal models was ﬁrst introduced in Multi-Instruct [36], where 47 diverse multi-modal tasks covering 11 broad categories were organized.”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, unknown): “” Table 1 presents a summary of our contributions in comparison to the recent works [32, 14, 17, 33, 34, 35] in the evaluation of multimodal chatbots.”
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, background): “The first branch [3, 19, 51] grates available multi-modal benchmark datasets and converts them to instruction format, with efforts like MultiIn-struct [51], InstructBLIP [3], and M 3 IT [19].”
A survey on multimodal large language models (2023, unknown): “For instance, MultiInstruct [26] proposes a metric called sensitivity that assesses the model’ robustness to varied instructions.”
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (2023, unknown): “Speciﬁcally, the ﬁrst manner [49, 18, 44] evaluates on existing traditional multimodal datasets, such as image caption [17] and VQA [23, 39, 35].”