Visual Instruction Tuning with Polite Flamingo

By Delong Chen, Jianfeng Liu, Wenliang Dai, Baoyuan Wang. 2023

56 citations recorded by the source corpus.

Recent research has demonstrated that the multi-task fine-tuning of multi-modal Large Language Models (LLMs) using an assortment of annotated downstream vision-language datasets significantly enhances their performance. Yet, during this process, a side effect, which we termed as the "multi-modal alignment tax", surfaces. This side effect negatively impacts the model's ability to format responses appropriately - for instance, its "politeness" - due to the overly succinct and unformatted nature of raw annotations, resulting in reduced human preference. In this paper, we introduce Polite Flamingo, a multi-modal response rewriter that transforms raw annotations into a more appealing, "polite" format. Polite Flamingo is trained to reconstruct high-quality responses from their automatically distorted counterparts and is subsequently applied to a vast array of vision-language datasets for response rewriting. After rigorous filtering, we generate the PF-1M dataset and further validate its value by fine-tuning a multi-modal LLM with it. Combined with novel methodologies including U-shaped multi-stage tuning and multi-turn augmentation, the resulting model, Clever Flamingo, demonstrates its advantages in both multi-modal understanding and response politeness according to automated and human evaluations. Code and dataset are available at https://github.com/ChenDelong1999/polite-flamingo

Subfields: Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Chunyuan Li
2. Haotian Liu
3. Yong Jae Lee
4. Yuheng Li
5. Saining Xie
6. Bo Li
7. Yuanhan Zhang
8. Ping Luo

Citations in the corpus

• Improved Baselines with Visual Instruction Tuning (2023, methodology): “We find that the inability [7] to balance between short-and long-form VQA for approaches like InstructBLIP [13], which leverages instruction following data that includes both natural responses and short-answers, is mainly due to the following reasons.”
• RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, methodology): “This multidataset tuning shares a similar spirit with recent studies on vision language learning, such as InstructBLIP [70], PaLI-X [71], and Clever Flamingo [72].”