Visual Programming: Compositional visual reasoning without training

By Tanmay Gupta, Aniruddha Kembhavi. 2022

718 citations recorded by the source corpus.

We present Visprog, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. Visprog avoids the need for any task-specific training. Instead, it uses the incontext learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing subroutines, or python functions to produce intermediate outputs that may be consumed by subsequent parts of the program. We demonstrate the flexibility of VIsPROG on 4 diverse tasks - compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. We believe neuro-symbolic approaches like Visprog are an exciting avenue to easily and effectively expand the scope of AI systems to serve the long tail of complex tasks that people may wish to perform.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Yoshua Bengio
  4. Percy Liang
  5. Luke S. Zettlemoyer
  6. Sergey Levine
  7. Quoc V. Le
  8. Caiming Xiong

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Trevor Darrell
  3. Dacheng Tao
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Cordelia Schmid

In Code Generation & Program Synthesis

  1. Graham Neubig
  2. Saining Xie
  3. Penghao Wu
  4. Carl Vondrick
  5. D'idac Sur'is
  6. Sachit Menon
  7. Yong Jae Lee
  8. Chunyuan Li

Citations in the corpus