A Unified Sequence Interface for Vision Tasks

By Ting-Li Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, Geoffrey E. Hinton. 2022

193 citations recorded by the source corpus.

While language tasks are naturally expressed in a single, unified, modeling framework, i.e., generating sequences of tokens, this has not been the case in computer vision. As a result, there is a proliferation of distinct architectures and loss functions for different vision tasks. In this work we show that a diverse set of"core"computer vision tasks can also be unified if formulated in terms of a shared pixel-to-sequence interface. We focus on four tasks, namely, object detection, instance segmentation, keypoint detection, and image captioning, all with diverse types of outputs, e.g., bounding boxes or dense masks. Despite that, by formulating the output of each task as a sequence of discrete tokens with a unified interface, we show that one can train a neural network with a single model architecture and loss function on all these tasks, with no task-specific customization. To solve a specific task, we use a short prompt as task description, and the sequence output adapts to the prompt so it can produce task-specific output. We show that such a model can achieve competitive performance compared to well-established task-specific models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Scalable Adaptive Computation for Iterative Generation (2022, methodology): “Self-conditioning for diffusion models was originally proposed in (Chen et al., 2022c).”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, methodology): “• Object detection is a generative object-detection task inspired by Chen et al. (2021; 2022).”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (2023, background): “Similarly, Pixel2Pixel [21] converted the outcome of visual comprehension tasks into a series of discrete tokens akin to language tasks.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “We encode sparse structures such as bounding boxes or human joints by adding 1000 special tokens to the vocabulary to represent discretized image coordinates (Chen et al., 2022b).”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “We formulate panoptic segmentation as a discrete data generation problem conditioned on pixels, similar to Pix2Seq [10, 11] but for dense prediction; hence we coin our approach Pix2Seq-D .”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, unknown): “Unified-IO [33], Pix2Seq v2 [8], and UniTab [63] extend this idea by using discrete coordinate tokens to encode and decode spatial information for more tasks.”
Segment Everything Everywhere All at Once (2023, methodology): “Compared with generalist models such as UViM [55], Pix2Seqv2 [56] and especially the recent model Painter [57] and SegGPT [50], our approach signiﬁcantly outperforms those methods on generic segmentation with a margin around 10 points on panoptic segmentation metrics.”
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding (2022, methodology): “Natural image understanding Pix2Seq uses the image-to-text architecture for core vision tasks such as object detection and instance segmentation (Chen et al., 2022b; 2021).”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[12] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey Hinton.”
Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning (2022, methodology): “For image captioning, following [11], we use MSCOCO 2017 captioning dataset [35].”
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, unknown): “Tasks: Inthesingle-armsetting, thereareﬁvetasks: 1)closing the drawer (CloseDrawer), 2) stacking the pink cube on top of the blue cube (StackCubes), 3) opening the lid of the box (Open-Box),4)placingatennisballintotheballbox,and5)uprightinga tipped-over mug (FlipMug).”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, unknown): “Chen et al. [116] reformulate and unify four core vision tasks (object detection, instance segmentation, key points prediction, and captioning) into a single pixel-to-sequence interface where both task description and outputs are converted into tokens.”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, methodology): “We create 1 , 000 bins, similar to [10,11,55,79], and represent regions using formats tailored to task requirements: – Box representation ( x 0 , y 0 , x 1 , y 1 ) : Utilized in tasks such as object detection and dense region captioning, with location tokens corresponding to the box coordinates.”