VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

By Jun Chen, Han Guo, Kai Yi, Boyang Albert Li, Mohamed Elhoseiny. 2021

299 citations recorded by the source corpus.

The limited availability of annotated data often hinders real-world applications of machine learning. To efficiently learn from small quantities of multimodal data, we leverage the linguistic knowledge from a large pre-trained language model (PLM) and quickly adapt it to new domains of image captioning. To effectively utilize a pretrained model, it is critical to balance the visual input and prior linguistic knowledge from pretraining. We propose VisualGPT, which employs a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the PLM with a small amount of in-domain image-text data. The proposed self-resurrecting activation unit produces sparse activations that prevent accidental overwriting of linguistic knowledge. When trained on 0.1%, 0.5% and 1% of the respective training sets, VisualGPT surpasses the best baseline by up to 10.0% CIDEr on MS COCO [43] and 17.9% CIDEr on Conceptual Captions [63]. Furthermore, VisualGPT achieves the state-of-the-art result on IU X-ray [15], a medical report generation dataset. Our code is available at https://github.com/Vision-CAIR/VisualGPT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Multimodal Few-Shot Learning with Frozen Language Models (2021, background): “Similarly, [46] and [6] show that a large pre-trained language model as decoder can improve a captioning performance when training data is limited.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “With the increased number of model parameters, it is hard to train those pretrained LLMs, thus more efforts have been paid to directly leverage the off-the-shelf frozen pre-trained LLMs for VL tasks [12, 38, 8, 46, 50].”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “Some methods freeze the language model to use the knowledge from LLMs for vision-to-language generation tasks (Tsimpoukelli et al., 2021; Alayrac et al., 2022; Chen et al., 2022a; Ma˜nas et al., 2023; Tiong et al., 2022; Guo et al., 2022).”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, background): “It dates back to at least Andreas et al. (2018) and saw renewed interest in the past few months due to the prevalence of large PLMs. Andreas et al. (2018) and Vong and Lake (2022) learn natural language descriptions that function as few-shot classiﬁers within an image-text matching model.”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, unknown): “Many case studies have been done for achieving effective and efﬁcient mobile AIGC networks as shown in Table III.”
Chameleon: Mixed-Modal Early-Fusion Foundation Models (2024, methodology): “Other models like LLaVA (Liu et al., 2023a), IDEFICS (Laurençon et al., 2023), and VisualGPT (Chen et al., 2022) also maintain separate image and text encoders.”
On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “…promising capabilities in text-to-image generation [55, 68, 72], image-grounded text generation (e.g., image captioning or visual question-answering) [2, 15, 42, 86], and joint generation [5, 32, 98] due to an increase in the amount of data, computational resources, and number of model parameters.”
ChatGPT is not all you need. A State of the Art Review of large Generative AI models (2023, methodology): “DALL · E 2 : DALL · E 2, created by OpenAI, is able to generate original, genuine and realistic images and art from a prompt consisting on a text description [10].”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “How can we combine the best of both the pre-trained LMs and image-text embedding models for visually grounded text generation? Existing supervised methods combine multimodal encoders by further training them on human-annotated paired image-text data [51, 6].”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, methodology): “For example, VisualGPT [119] utilizes linguistic knowledge from large language models and adapts it to new domains of image captioning in an efficient manner, even with small amounts of multimodal data.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, methodology): “[13] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny.”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, methodology): “…techniques in EarthGPT are elaborated, involving the visual-enhanced perception mechanism (see Section III-B), the language perception (see Section III-C), the cross-modal mutual comprehension approach (see Section III-D), and the unified instruction tuning method for RS (see Section III-E).”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “Based on different similarity functions, existing retrieval methods can be categorized into sparse retrieval, dense retrieval, and others.”
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (2023, unknown): “With the remarkable generalization abilities of LLMs, interesting studies have extended LLMs to multi-modal domains by aligning visual inputs with LLMs. Early works such as VisualGPT [5] and Frozen [51] used pre-trained language models to improve vision-language models on image captioning and visual”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, methodology): “For example, models like Uniter [18], Oscar [19], VinVL [20], and LiT [21] freeze the image encoder, while Frozen [22] and VGPT [23] freeze the language model.”
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2022, background): “, 2019)) and fine-tune these models with image-caption pairs (Chen et al., 2021; Mokady et al., 2021; Luo et al., 2022).”