Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training

By Anthony Meng Huat Tiong, Junnan Li, Boyang Albert Li, Silvio Savarese, Steven C. H. Hoi. 2022

141 citations recorded by the source corpus.

Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PNP-VQA), a modular framework for zero-shot VQA. In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality, PNP-VQA requires no additional training of the PLMs. Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captions, and pass the captions to a PLM as context for question answering. Surpassing end-to-end trained baselines, PNP-VQA achieves state-of-the-art results on zero-shot VQAv2 and GQA. With 11B parameters, it outperforms the 80B-parameter Flamingo model by 8.5% on VQAv2. With 738M PLM parameters, PNP-VQA achieves an improvement of 9.1% on GQA over FewVLM with 740M PLM parameters. Code is released at https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

ViperGPT: Visual Inference via Python Execution for Reasoning (2023, neutral): “TRiG [13] 50.5 KAT [16] 54.4 RA-VQA [32] 54.5 REVIVE [33] 58.0 PromptCap [21] 58.8 Z S PNP-VQA [52] 35.9 PICa [60] 43.3 BLIP-2 [30] 45.9 Flamingo [1] 50.6 ViperGPT (ours) 51.9”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “Some methods freeze the language model to use the knowledge from LLMs for vision-to-language generation tasks (Tsimpoukelli et al., 2021; Alayrac et al., 2022; Chen et al., 2022a; Mañas et al., 2023; Tiong et al., 2022; Guo et al., 2022).”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Techniques for combining pretrained models include joint finetuning on a new task [4, 33, 40, 67], communication through prompting [62,69], composing probability distributions of energy-based models [11, 37], guiding one model with feedback from another [61], and iterative optimization [34].”
Modular Visual Question Answering via Code Generation (2023, methodology): “Our implementation of this function is based on PnP-VQA (Tiong et al., 2022) and PICa (Yang et al.”
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (2023, background): “In recent years, the trend of using autoregressive language models as decoders in vision-language tasks has gained significant traction [6, 15, 36, 31, 2, 16, 17, 12].”
On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “The knowledge contained within these powerful LMs is used to facilitate vision-language tasks [26, 33, 84, 93, 101].”
Multimodal Large Language Models: A Survey (2023, methodology): “PNP-VQA [48] utilizes a caption model and pre-trained language model (PLM) to address the VQA task.”
Declarative Knowledge Distillation from Large Language Models for Visual Question Answering Datasets (2024, methodology): “Approaches that use these VLMs as components are, e.g., ViperGPT (Sur´ıs, Menon, and Vondrick 2023), CodeVQA (Subramanian et al. 2023), and PnP-VQA (Tiong et al. 2022).”