ViperGPT: Visual Inference via Python Execution for Reasoning

By D'idac Sur'is, Sachit Menon, Carl Vondrick. 2023

823 citations recorded by the source corpus.

Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ${\color{green}{\text{ViperGPT}}}$, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ${\color{green}{\text{ViperGPT}}}$ utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Percy Liang
  4. Luke S. Zettlemoyer
  5. Yoshua Bengio
  6. Quoc V. Le
  7. Yejin Choi
  8. Graham Neubig

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Xiaogang Wang
  4. Trevor Darrell
  5. Ming-Hsuan Yang
  6. Yu Qiao
  7. Jitendra Malik
  8. Andrew Zisserman

In Prompting & Reasoning

  1. Hongsheng Li
  2. Peng Gao
  3. Renrui Zhang
  4. Pan Lu
  5. Luke S. Zettlemoyer
  6. Yu Qiao
  7. Ziwei Liu
  8. Ziyi Lin

In Code Generation & Program Synthesis

  1. Graham Neubig
  2. Yong Jae Lee
  3. Chunyuan Li
  4. Haotian Liu
  5. Qingyang Wu
  6. Noah A. Smith
  7. Percy Liang
  8. Hao Peng

Citations in the corpus