A Generalist Framework for Panoptic Segmentation of Images and Videos

By Ting-Li Chen, Lala Li, Saurabh Saxena, Geoffrey E. Hinton, David J. Fleet. 2022

146 citations recorded by the source corpus.

Panoptic segmentation assigns semantic and instance ID labels to every pixel of an image. As permutations of instance IDs are also valid solutions, the task requires learning of high-dimensional one-to-many mapping. As a result, state-of-the-art approaches use customized architectures and task-specific loss functions. We formulate panoptic segmentation as a discrete data generation problem, without relying on inductive bias of the task. A diffusion model is proposed to model panoptic masks, with a simple architecture and generic loss function. By simply adding past predictions as a conditioning signal, our method is capable of modeling video (in a streaming setting) and thereby learns to track object instances automatically. With extensive experiments, we demonstrate that our simple approach can perform competitively to state-of-the-art specialist methods in similar settings. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ping Luo
2. Allan Jabri
3. David J. Fleet
4. Ming-Hsuan Yang
5. Ting-Li Chen
6. Pei Sun
7. Yibing Song
8. Shoufa Chen

In Computer Vision and Pattern Recognition

1. Ming-Hsuan Yang
2. Luc Van Gool
3. Dacheng Tao
4. Xiaogang Wang
5. Jitendra Malik
6. Trevor Darrell
7. Andrew Zisserman
8. Andrea Vedaldi

In Model Architectures

1. Ming-Hsuan Yang
2. Junhwa Hur
3. Varun Jampani
4. Allan Jabri
5. Charles Herrmann
6. Ting-Li Chen
7. David J. Fleet
8. Pei Sun

Citations in the corpus

• Scalable Adaptive Computation for Iterative Generation (2022, methodology): “Self-conditioning for diffusion models was originally proposed in (Chen et al., 2022c).”
• CogVLM: Visual Expert for Pretrained Language Models (2023, unknown): “Many vision and cross-modality tasks can be formulated as next token prediction, e.g., image captioning [Agrawal et al., 2019], visual question answering [Antol et al., 2015], visual grounding [Yu et al., 2016] and even segmentation [Chen et al., 2022a].”
• DiffusionDet: Diffusion Model for Object Detection (2022, methodology): “Some pioneer works tried to adopt the diffusion model for image segmentation tasks [1, 5, 6, 12, 28, 42, 89], for example, Chen et al.”
• TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation (2024, unknown): “However, using tokenization for continuous or high-dimensional data has proven to be extremely challenging for training [31], requires a huge amount of data [32], [33], and tends to converge to a single state [34].”
• A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence (2023, background): “, image segmentation [3, 4, 8, 24, 55, 64], object detection [7], and monocular depth estimation [13, 50].”