UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes

By Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, Neil Houlsby. 2022

84 citations recorded by the source corpus.

We introduce UViM, a unified approach capable of modeling a wide range of computer vision tasks. In contrast to previous models, UViM has the same functional form for all tasks; it requires no task-specific modifications which require extensive human expertise. The approach involves two components: (I) a base model (feed-forward) which is trained to directly predict raw vision outputs, guided by a learned discrete code and (II) a language model (autoregressive) that is trained to generate the guiding code. These components complement each other: the language model is well-suited to modeling structured interdependent data, while the base model is efficient at dealing with high-dimensional outputs. We demonstrate the effectiveness of UViM on three diverse and challenging vision tasks: panoptic segmentation, depth prediction and image colorization, where we achieve competitive and near state-of-the-art results. Our experimental results suggest that UViM is a promising candidate for a unified modeling approach in computer vision.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “Other closely related models include UViM (Kolesnikov et al., 2022) which generates a discrete guiding code for a D-VAE to build an autoregressive model for panoptic segmentation, depth prediction, and colorization, and Pix2Seq v2 (Chen et al.”
Autoregressive Image Generation without Vector Quantization (2024, background): “Vector-quantized tokenizers are difficult to train and are sensitive to gradient approximation strategies [51, 41, 40, 27].”
Segment Everything Everywhere All at Once (2023, methodology): “Compared with generalist models such as UViM [55], Pix2Seqv2 [56] and especially the recent model Painter [57] and SegGPT [50], our approach signiﬁcantly outperforms those methods on generic segmentation with a margin around 10 points on panoptic segmentation metrics.”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, unknown): “UVim (Kolesnikov et al., 2022) is another unified approach for vision that uses a language model to generate the guiding code for a second model to predict raw vision outputs.”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (2023, methodology): “Subsequently, we employ LoRA [38] to train an open-source LLM on the collected instruction dataset, thus adapting the LLM to use tools. while UViM [28] and Unified-IO [20] advocated for the learned discrete codes as a means to unify an array of vision tasks.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “[44] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby.”
Finite Scalar Quantization: VQ-VAE Made Simple (2023, methodology): “We show that FSQ can serve as a drop-in replacement for VQ in various architectures, for different datasets and tasks, by applying it to MaskGIT (Chang et al., 2022) for image generation, and in UViM (Kolesnikov et al., 2022) for depth estimation, colorization, and panoptic segmentation.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Our work is also related to recent generalist vision models [10, 11, 60, 31, 43] where both architecture and loss functions are task-agnostic.”
A Unified Sequence Interface for Vision Tasks (2022, methodology): “Concurrently to our work, Gato [36] uniﬁes a series of vision and control tasks into a single sequential prediction problem, and UViM [21] and Uniﬁed-IO [31] propose using learned discrete codes for unifying a set of vision tasks.”