Visual Prompting via Image Inpainting

By Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, Alexei A. Efros. Neural Information Processing Systems, 2022

324 citations recorded by the source corpus.

How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification? Inspired by prompting in NLP, this paper investigates visual prompting: given input-output image example(s) of a new task at test time and a new input image, the goal is to automatically produce the output image, consistent with the given examples. We show that posing this problem as simple image inpainting - literally just filling in a hole in a concatenated visual prompt image - turns out to be surprisingly effective, provided that the inpainting algorithm has been trained on the right data. We train masked auto-encoders on a new dataset that we curated - 88k unlabeled figures from academic papers sources on Arxiv. We apply visual prompting to these pretrained models and demonstrate results on various downstream image-to-image tasks, including foreground segmentation, single object detection, colorization, edge detection, etc.

Subfields: Biophysics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0a25c137edc7c9752aa6d99ae4084683c3fe6b56

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

1. Chunhua Shen
2. Carl Vondrick
3. Yue Cao
4. Hongsheng Li

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Chunhua Shen
3. Xiaogang Wang
4. Luc Van Gool
5. Dacheng Tao
6. Yu Qiao
7. Yi Yang
8. Alan L. Yuille

Citations in the corpus

• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, unknown): “To tackle image inpainting tasks, Bar et al. [3] introduces the first visual prompting framework that utilizes inpainting with discrete tokens on images.”
• A Survey on In-context Learning (2022, methodology): “9.1 Visual In-Context Learning Bar et al. (2022) employ an image patch infill-ing task in grid-like images using masked auto-encoders (MAE) to train their model.”
• Personalize Segment Anything Model with One Shot (2023, background): “• Target-semantic Prompting.”
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, unknown): “LLama [63, 64] release a collection of pre-trained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.”
• Images Speak in Images: A Generalist Painter for In-Context Visual Learning (2022, background): “While the work [6] proves the concept of in-context learning for vision tasks, no results were reported on standard benchmark datasets.”
• Doubly Right Object Recognition: A Why Prompt for Visual Rationales (2022, background): “Recently, the computer vision field has adapted the language and proposed the visual prompts to adapt vision models [4, 5, 21, 43].”