Generalized Decoding for Pixel, Image, and Language

By Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Singh Behl, Jianfeng Wang, Lu Yuan and 4 more. 2022

380 citations recorded by the source corpus.

We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decoder takes as input two types of queries: (i) generic non-semantic queries and (ii) semantic queries induced from text inputs, to decode different pixel-level and token-level outputs in the same semantic space. With such a novel design, X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks. Without any pseudo-labeling, our design enables seamless interactions across tasks at different granularities and brings mutual benefits by learning a common and rich pixel-level understanding. After pretraining on a mixed set of a limited amount of segmentation data and millions of image-text pairs, X-Decoder exhibits strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings. Notably, it achieves (1) state-of-the-art results on open-vocabulary segmentation and referring segmentation on seven datasets; (2) better or competitive finetuned performance to other generalist and specialist models on segmentation and VL tasks; and (3) flexibility for efficient fine-tuning and novel task composition (e.g., referring captioning and image editing shown in Fig. 1). Code, demo, video and visualization are available at: https://x-decoder-vl.github.io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Visual Instruction Tuning (2023, methodology): “( ii ) A system that coordinates various models via LangChain [1] / LLMs [35], such as Visual ChatGPT [53], X-GPT [63], MM-REACT [55], VisProg [18], and ViperGPT [46].”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Note that the results of GLIPv2 and X-Decoder on COCO are fine-tuned while those of OpenSeeD are reported without fine-tuning.”
GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, background): “In contrast, in the recognition field, the long-standing paradigm has been to build recognition models [29, 37, 76] by starting from a foundation model pretrained on large-scale image data [4,15,16] or image-text pairs [30, 44, 68].”
MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023, methodology): “These approaches [48, 39, 11, 57, 57, 52] take an LLM as a dispatch scheduler and connect different expert models through it to allow for different tasks.”
LISA: Reasoning Segmentation via Large Language Model (2023, result): “Further, we also find that with the Mask2Former backbone, our framework still achieves a decent performance on the reasoning segmentation task, significantly outperforming previous works such as X-Decoder [65].”
Segment Everything Everywhere All at Once (2023, methodology): “Our model framework follows X-Decoder [11] except the decoder.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Recent studies [237], [361], [362], [363] demonstrate the potential of a universal transformer architecture that concurrently learns segmentation alongside visual language tasks, paving the way for integrated multi-modal segmentation learning.”
SVIT: Scaling up Visual Instruction Tuning (2023, background): “Existing multimodal solutions can be roughly split into two categories: 1) multimodal systems, e.g. Visual ChatGPT (Wu et al., 2023), X-Decoder (Zou et al., 2023) and In-ternGPT (Liu et al., 2023d), in which multiple language and vision models are coordinated by a LLM manager/controller to deal…”
Segment Anything in High Quality (2023, methodology): “We follow the same inference pipeline of SAM but use the mask prediction from HQ-Output token as high-quality mask prediction.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023, methodology): “Works in this category include VisualChatGPT [35], HuggingGPT [29], Cola [8], X-GPT [42], MM-REACT [37], and ViperGPT [31].”