ViperGPT: Visual Inference via Python Execution for Reasoning
By D'idac Sur'is, Sachit Menon, Carl Vondrick. 2023
823 citations recorded by the source corpus.
Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ${\color{green}{\text{ViperGPT}}}$, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ${\color{green}{\text{ViperGPT}}}$ utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Quoc V. Le
- Yejin Choi
- Graham Neubig
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Yu Qiao
- Jitendra Malik
- Andrew Zisserman
In Prompting & Reasoning
In Code Generation & Program Synthesis
Citations in the corpus
- Visual Instruction Tuning (2023, methodology): “( ii ) A system that coordinates various models via LangChain [1] / LLMs [35], such as Visual ChatGPT [53], X-GPT [63], MM-REACT [55], VisProg [18], and ViperGPT [46].”
- LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “In recent years, with the continual expansion of LLMs’ capabilities, there has been a growing trend [14, 18, 21, 27, 37, 42, 51, 58, 61, 67, 69, 79] to combine them with visual foundation models and leverage their combined strengths to tackle more complex visionlanguage tasks.”
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, methodology): “A recent line of work relies on large amounts of supervision [43, 21] and focuses on generating commands [34] and programs [45, 11] to infer the choice of tools.”
- MM-LLMs: Recent Advances in MultiModal Large Language Models (2024, neutral): “, DocLLM( I D ) (Wang et al., 2023a), LLaVA-ϕ (Zhu et al., 2024c), Yi-VL(Team., 2023 Gemini (Team et al., 2023), ModaVerse (Wang et al., 2024c), MLLM-Tool(Wang et al., 2024a) Design Division Tool-using Visual ChatGPT(Wu et al., 2023a), ViperGPT(Surís et al., 2023), MM-REACT(Yang et al., 2023b), Hugg”
- Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “Recent neuro-symbolic methods like visual programming [20, 26, 73] also use LLMs like ChatGPT to decompose complex visual tasks (described in natural language) into modular components.”
- A survey on large language model based autonomous agents (2023, positive): “ViperGPT [128] leverages models like GLIP to extract image features for visual content-related actions.”
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (2024, background): “…image input and text prompts, providing a general, expressive interface for a growing spectrum of applications – grounded chat (Li et al., 2023c; Gong et al., 2023), visual programming (Sur’is et al., 2023; Sub-ramanian et al., 2023), robotic control (Driess et al., 2023; Brohan et al., 2023), etc.”
- A Comprehensive Overview of Large Language Models (2023, neutral): “Neither of these details are important as others to mention for instruction-tuned models nor provided by the papers.”
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “These methods are exemplified by VisualChatGPT [60], HuggingGPT [52], MM-REACT [65], and ViperGPT [54].”
- PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, background): “ViperGPT [34] is a pioneering example, harnessing code-producing LLMs, including the likes of GPT-3 Codex [4].”
- TempCompass: Do Video LLMs Really Understand Videos? (2024, background): “The Pipeline paradigm (Shen et al., 2023; Sur’is et al., 2023; Wu et al., 2023; Yang et al., 2023) leverages off-the-shelf vision expert models to extract visual information in the form of texts, which are then fed to LLMs to perform the downstream vision tasks.”
- Executable Code Actions Elicit Better LLM Agents (2024, methodology): “The following is an example tool definition for web-browsing.”
- Modular Visual Question Answering via Code Generation (2023, methodology): “For CodeVQA and Few-shot VQA, we used 7 in-context examples to be consistent with the OK-VQA results of ViperGPT (Surís et al., 2023).”
- GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest (2023, neutral): “Some representative works are Visual ChatGPT (Wu et al., 2023), ViperGPT (Surís et al., 2023), MM-REACT (Yang et al., 2023c), InternGPT (Liu et al., 2023d), VideoChat (Li et al., 2023b), etc.”
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “…difficulties, a line of recent works in building multimodal systems has leveraged pre-trained LLMs, with some augmenting with new modality encoders [5, 46, 119], some adding modality specific decoders [14, 96] and others leveraging the LLM’s capabilities to build modular frameworks [64, 166, 173].”
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (2023, methodology): “ViperGPT [29] demonstrates the potential of combining an LLM with different vision models to address complex visual queries programmatically.”
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (2023, methodology): “The recent development of multi-modal large language models (MLLMs) [15] greatly improves the performance of complex reasoning on images, image-based dialogue, and language grounding tasks [8, 6, 13, 14, 15].”