Images Speak in Images: A Generalist Painter for In-Context Visual Learning

By Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, Tiejun Huang. Computer Vision and Pattern Recognition, 2022

387 citations recorded by the source corpus.

In-context learning, as a new paradigm in NLP, allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. But in computer vision, the difficulties for in-context learning lie in that tasks vary significantly in the output representations, thus it is unclear how to define the general-purpose task prompts that the vision model can understand and transfer to out-of-domain tasks. In this work, we present Painter, a generalist model which addresses these obstacles with an “image”-centric solution, that is, to redefine the output of core vision tasks as images, and specify task prompts as also images. With this idea, our training process is extremely simple, which performs standard masked image modeling on the stitch of input and output image pairs. This makes the model capable of performing tasks conditioned on visible image patches. Thus, during inference, we can adopt a pair of input and output images from the same task as the input condition, to indicate which task to perform. Without bells and whistles, our generalist Painter can achieve competitive performance compared to well-established task-specific models, on seven representative vision tasks ranging from high-level visual understanding to low-level image processing. In addition, Painter significantly outperforms recent generalist models on several challenging tasks.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9ceaeff7117965832f4c05fd6355d021862d0a82

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Generative Multimodal Models are In-Context Learners (2023, background): “Painter [78] and SegGPT [79] conduct an early study of visual in-context learning.”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, neutral): “…to overcome the limitations imposed by pre-defined tasks, resulting in a gap in open-ended task capabilities compared to LLMs. Recently, visual prompt tuning [24, 66, 70, 67, 54] has emerged as a way to flexibly outline some pure vision tasks (see Figure 1b), such as object detection, instance”
The rise and potential of large language model based agents: a survey (2025, positive): “Beyond texts, researchers also explore the potential ICL capabilities in diﬀerent multimodal tasks [189–194], making it possible for agents to be applied to large-scale real-world tasks.”
Point Transformer V3: Simpler, Faster, Stronger (2023, background): “Deep learning models have experienced rapid advancements in various areas, such as 2D vision [24, 38, 78, 86] and natural language processing (NLP) [1, 37, 56, 79], with their progress often attributed to the effective utilization of scale, encompassing factors such as the size of datasets, the…”
A Survey on In-context Learning (2022, methodology): “Wang et al. (2023f) propose to utilize an extra text prompt to guide a generative model in comprehensively producing the desired image.”
Segment Everything Everywhere All at Once (2023, methodology): “Compared with generalist models such as UViM [55], Pix2Seqv2 [56] and especially the recent model Painter [57] and SegGPT [50], our approach significantly outperforms those methods on generic segmentation with a margin around 10 points on panoptic segmentation metrics.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, neutral): “Innovations like Painter [69] and LVM [5] have leveraged visual prompts to design in-context learning paradigms, thereby facilitating the generalization to downstream unseen tasks.”
OmniGen: Unified Image Generation (2024, neutral): “Additionally, due to resource constraints, the number of input images during training is limited to a maximum of three, preventing the model from handling long image sequences.”