Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

By Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello. 2023

479 citations recorded by the source corpus.

We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. Text-to-image diffusion models have the remarkable ability to generate high-quality images with diverse open-vocabulary language descriptions. This demonstrates that their internal representation space is highly correlated with open concepts in the real world. Text-image discriminative models like CLIP, on the other hand, are good at classifying images into open-vocabulary labels. We leverage the frozen internal representations of both these models to perform panoptic segmentation of any category in the wild. Our approach outperforms the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks. In particular, with COCO training only, our method achieves 23.4 PQ and 30.0 mIoU on the ADE20K dataset, with 8.3 PQ and 7.9 mIoU absolute improvement over the previous state of the art. We open-source our code and models at https://github.com/NVlabs/ODISE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Compared with state-of-the-art methods ODISE [47] and X-Decoder [61], OpenSeeD achieves 1.1 and 1.9 mask AP improvements on ADE20K, respectively.”
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, methodology): “Furthermore, since these models can generate specific scenes and have some degree of control over the layout and placement of objects [69], they should be able to localize objects and were shown in [65] to be effective for semantic segmentation.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Recently, ODISE [286] uses a frozen diffusion model as the feature extractor, a Mask2Former head, and joint training with caption data to perform open vocabulary panoptic segmentation.”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “• In addition, since the diffusion model is used as the image decoder, it would be interesting to explore DMs-based zero-shot perception such as classification (Li et al., 2023a) and segmentation (Xu et al., 2023a) with the goal of unifying all perception tasks with image synthesis/generation.”
LangSplat: 3D Language Gaussian Splatting (2023, methodology): “We observe that our method not only out-performs 2D-based methods such as ODISE [41] and OV-Seg [22], but also achieves better results than 3D-based methods including LERF [18] and 3D-OVS [23] by a large margin.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “ODISE [295] explores diffusion models for open-vocabulary segmentation tasks, and proposes a novel implicit captioner to generate captions for images for better utilizing pre-trained large-scale text-to-image diffusion models.”
Emergent Correspondence from Image Diffusion (2023, background): “Some recent works [5, 90, 96] fine-tune either the U-Net or the attached head for dense prediction tasks and yield better performance.”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, background): “As large vision-language models gained popularity for image classification, numerous new approaches [11, 20, 23, 26, 32, 40, 43, 45, 48, 56, 71, 72, 79, 80] have emerged to tackle open-vocabulary or zero-shot image semantic segmentation.”
Segment Everything Everywhere All at Once (2023, methodology): “Additionally, ODISE [52] leverages a text-to-image diffusion model as the backbone for open-vocabulary segmentation.”