OneFormer: One Transformer to Rule Universal Image Segmentation

By Jitesh Jain, Jiacheng Li, M. Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi. 2022

605 citations recorded by the source corpus.

Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better intertask and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, Cityscapes, and COCO, despite the latter being trained on each task individually. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Segment Anything (2023, background): “In a multi-task system, a single model performs a ﬁxed set of tasks, e.g ., joint semantic, instance, and panoptic segmentation [114, 19, 54], but the training and test tasks are the same.”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Recently, Detection Transformer (DETR) [1] that is based on Transformer [44] has achieved significant progress in many detection [28, 34, 37, 55, 60] and segmentation models [5, 18, 29, 32].”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, methodology): “The state-of-the-art segmentation method OneFormer [35] achieves an Intersection-over-Union (IoU) with 0.58 on the ground-truth set.”
Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models (2023, background): “Cross-Attentions [62] are widely adopted deep learning modules used in discriminative models [20, 6, 29], conditional generation models [52, 57, 54] as well as language models [17, 51, 14].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Recent studies [26], [126], [129], [163], [172], [173], [271] have employed query-based transformers for various segmentation tasks within a uniﬁed architecture.”
ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (2024, methodology): “…our method freezes all trained parameters and fine-tunes only a small set of prompt embeddings, robustly keeping the previous knowledge and extending the scalability of the model. leveraging convolutional neural networks [7, 23, 43, 48] and transformer-based architectures [1, 8, 9, 18, 26].”
Segment Everything Everywhere All at Once (2023, methodology): “Recently, the Detection Transformer (DETR)[4], a model based on the Transformer[47] architecture, has made significant advances in segmentation [29, 8, 25, 60, 19] tasks.”
Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, positive): “Query-based seg-menters [9, 10, 21, 55] have unified image segmentation via mask classification and solve semantic, instance, and panoptic segmentation using the same framework.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, neutral): “Segmenter [57] ViT-L [16] 51.8 SegFormer [69] MiT-B5 [69] 51.0 Mask2Former [12] Swin-L [38] 56.4 UperNet [68] BEiT-L [2] 56.3 ViT-Adapter [11] BEiT-L [2] 58.3 OneFormer [24] Swin-L [38] 57.4 OneFormer [24] ConNeXt-XL [40] 57.4 Ours ViT-L [16] 59.4 Table 8.”