Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation

By Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel Ming-shuan Ni, Harry Shum. 2022

638 citations recorded by the source corpus.

In this paper we present Mask DINO, a unified object detection and segmentation framework. Mask DINO extends DINO (DETR with Improved Denoising Anchor Boxes) by adding a mask prediction branch which supports all image segmentation tasks (instance, panoptic, and semantic). It makes use of the query embeddings from DINO to dot-product a high-resolution pixel embedding map to predict a set of binary masks. Some key components in DINO are extended for segmentation through a shared architecture and training process. Mask DINO is simple, efficient, and scalable, and it can benefit from joint large-scale detection and segmentation datasets. Our experiments show that Mask DINO significantly outperforms all existing specialized segmentation methods, both on a ResNet-50 backbone and a pre-trained model with SwinL backbone. Notably, Mask DINO establishes the best results to date on instance segmentation (54.5 AP on COCO), panoptic segmentation (59.4 PQ on COCO), and semantic segmentation (60.8 mIoU on ADE20K) among models under one billion parameters. Code is available at https://github.com/IDEA-Research/MaskDINO.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Mask DINO is a unified detection and segmentation framework which simultaneously predicts box and mask.”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, methodology): “Motivated by the encouraging progress of Transformerbased detectors [24, 25, 31, 58], in this work, we propose to build a strong open-set detector based on DINO [58], which not only offers the state-of-the-art object detection performance, but also allows us to integrate multi-level text information”
Gaussian Grouping: Segment and Edit Anything in 3D Scenes (2023, methodology): “3D object style transfer results comparison between Instruct-NeRF2NeRF [10] and our Gaussian Grouping under the same instruction prompt “Turn the bear into a panda” using InstructPix2Pix [2].”
DETRs with Hybrid Matching (2022, background): “…have achieved signiﬁcant progress on various fundamental vision recognition tasks such as object detection [74, 81, 44, 52], instance segmentation [23, 12, 11, 70], panoptic segmentation [8, 29, 58, 69, 71], referring expression segmentation [63, 67], video instance segmenta-† Equal contribution.”
Vision Transformers are Good Mask Auto-Labelers (2023, background): “DINO [9] further shows that meaning visual grouping emerges during self-supervised learning with ViTs.”
Planning-oriented Autonomous Driving (2022, background): “It could be simply drawn via matrix multiplication, as in recent query-based segmentation works [11, 29].”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, background): “Work on this problem has led to significant improvements along several threads including image level predictions like image classification [13, 54, 58], object level predictions like object detection [33,51,53,59–61], as well as pixel level predictions like semantic segmentation [10,29,51,53].”
ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (2024, methodology): “…our method freezes all trained parameters and fine-tunes only a small set of prompt embeddings, robustly keeping the previous knowledge and extending the scalability of the model. leveraging convolutional neural networks [7, 23, 43, 48] and transformer-based architectures [1, 8, 9, 18, 26].”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “As we can see, when using regular training settings for fair comparison, the detection performance of ViT (Li et al., 2021b) and ViTDet (Li et al., 2022b) is inferior to recent vision-speciﬁc models.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Recent work has proposed end-to-end approaches with Transformer based architectures [58, 17, 70, 16, 35, 68, 69, 33], for which the model directly predicts segmentation masks and optimizes based on a bipartite graph matching loss.”
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks (2023, result): “We compare with previous strong models, including FD-SwinV2 [64], DINO [72], Mask DINO [29], FDCLIP [64], CoCa [69], OFA [61], Florence [70].”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, background): “(timestamp: Nov 10, 2022) methods / results reference. a: BEiT-3 [104], b: iBOT [130], c: Open CLIP-H [48], d: Text4Vis [112], e: MaskFeat [106], f: Group DETRv2 [19], g: FocalNet [119], h: FD-SwinV2-G [110], i: Mask DINO [57], j: LVIS 2021 competition 1 st [36], k: ViT-Adapter [23]. pure MIM pre-tr”