Visual Programming: Compositional visual reasoning without training
By Tanmay Gupta, Aniruddha Kembhavi. 2022
718 citations recorded by the source corpus.
We present Visprog, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. Visprog avoids the need for any task-specific training. Instead, it uses the incontext learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing subroutines, or python functions to produce intermediate outputs that may be consumed by subsequent parts of the program. We demonstrate the flexibility of VIsPROG on 4 diverse tasks - compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. We believe neuro-symbolic approaches like Visprog are an exciting avenue to easily and effectively expand the scope of AI systems to serve the long tail of complex tasks that people may wish to perform.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Code Generation & Program Synthesis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Luke S. Zettlemoyer
- Sergey Levine
- Quoc V. Le
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Code Generation & Program Synthesis
- Graham Neubig
- Saining Xie
- Penghao Wu
- Carl Vondrick
- D'idac Sur'is
- Sachit Menon
- Yong Jae Lee
- Chunyuan Li
Citations in the corpus
- Visual Instruction Tuning (2023, methodology): “( ii ) A system that coordinates various models via LangChain [1] / LLMs [35], such as Visual ChatGPT [53], X-GPT [63], MM-REACT [55], VisProg [18], and ViperGPT [46].”
- ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “Concurrent work [11] generates a list of pseudocode instructions and interprets them as a ‘visual program,’ relying on in-context learning from provided examples.”
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, methodology): “A recent line of work relies on large amounts of supervision [43, 21] and focuses on generating commands [34] and programs [45, 11] to infer the choice of tools.”
- Evaluating Text-to-Visual Generation with Image-to-Text Generation (2024, methodology): “Recent neuro-symbolic methods like visual programming [20, 26, 73] also use LLMs like ChatGPT to decompose complex visual tasks (described in natural language) into modular components.”
- A Comprehensive Overview of Large Language Models (2023, neutral): “Serving as a controller [229], decision maker [233], or”
- LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, background): “In recent years, with the continual expansion of LLMs’ capabilities, there has been a growing trend [14, 18, 21, 27, 37, 42, 51, 58, 61, 67, 69, 79] to combine them with visual foundation models and leverage their combined strengths to tackle more complex visionlanguage tasks.”
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (2023, methodology): “The recent development of multi-modal large language models (MLLMs) [15] greatly improves the performance of complex reasoning on images, image-based dialogue, and language grounding tasks [8, 6, 13, 14, 15].”
- V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “For the LLM-tool-using systems, we evaluate methods including MM-React [53], VisualChat-GPT [54], and Visprog [12].”
- Modular Visual Question Answering via Code Generation (2023, background): “, 2023), VisProg (Gupta and Kembhavi, 2023), and ViperGPT (Surís et al.”
- VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, neutral): “One approach focuses on leveraging existing pretrained task-specific tools such as image captioners [15], object detectors [59] and text-to-image generators [60] by either creating multimodal prompt interfaces [43, 61] or by executing LLM-generated programs [62, 63, 64].”
- LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, methodology): “VisProg [19] and ViperGPT [44] uses LLMs to design modular pseudocode instructions or executable Python programs to achieve visual reasoning.”
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “…difficulties, a line of recent works in building multimodal systems has leveraged pre-trained LLMs, with some augmenting with new modality encoders [5, 46, 119], some adding modality specific decoders [14, 96] and others leveraging the LLM’s capabilities to build modular frameworks [64, 166, 173].”
- A survey on multimodal large language models (2023, positive): “…as instructions and previous reasoning history [14,65,75,76,82,83] Training-free Few-shot Chameleon [75], HuggingGPT [80], MM-REACT [14], VISPROG [76] Zero-shot CAT [84], CaFo [90], Visual ChatGPT [83], IdealGPT [91], ChatCaptioner [92], PointCLIP V2 [93], SMs [94] Finetuning GPT4Tools [34]…”
- LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, positive): “VISPROG [19] positions the LLM as a decision-making agent, enhancing visual reasoning by invoking task-specific visual modules.”
- VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, neutral): “• Multi-step Visual Program [136, 137]: Given a task “open the drawer precisely by half” where there is insufficient information because object models are not available, VoxPoser can come up with multi-step manipulation strategies based on visual feedback that first opens the drawer fully while reco”
- DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “One line of work is built by system integration of LLMs with various functioning agents where language acts as general interface (Wu et al., 2023; Gupta & Kembhavi, 2023; Yang et al., 2023b; Liang et al., 2023; Shen et al., 2023; Yang et al., 2023a; Surís et al., 2023), and remarkable success has…”
- Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, methodology): “Similar to recent works like ViperGPT and VPD [14, 38, 18], S KETCHPAD enables LMs to sketch through code generation.”
- Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Recently, Visual Programming uses the inter-pretable neuro-symbolic representation to break down a complex task into more straightforward steps that explain the rationale of a particular output from GPT-3 [217].”