CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation

By Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell D. Collins, Yukun Zhu, Hartwig Adam, Alan L. Yuille, Liang-Chieh Chen. 2022

112 citations recorded by the source corpus.

We propose Clustering Mask Transformer (CMT-DeepLab), a transformer-based framework for panoptic segmentation designed around clustering. It rethinks the existing transformer architectures used in segmentation and detection; CMT-DeepLab considers the object queries as cluster centers, which fill the role of grouping the pixels when applied to segmentation. The clustering is computed with an alternating procedure, by first assigning pixels to the clusters by their feature affinity, and then updating the cluster centers and pixel features. Together, these operations comprise the Clustering Mask Transformer (CMT) layer, which produces cross-attention that is denser and more consistent with the final segmentation task. CMT-DeepLab improves the performance over prior art significantly by 4.4% PQ, achieving a new state-of-the-art of 55.7% PQ on the COCO test-dev set.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

OneFormer: One Transformer to Rule Universal Image Segmentation (2022, background): “CMT-DeepLab‡ [59] MaX-S† [50] — — — 1025×2049 60k 64.”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, unknown): “For image understanding tasks ( e.g ., image classification [17], object detection [8, 77, 74], segmentation [64, 70, 72], and Multi-modal Large Language Models (MLLMs) [1, 40, 73]), it is common to use a general feature encoder instead of an autoen-coder to tokenize the image.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “To handle this, CMT-Deeplab [227] proposes an alternating procedure for object query and decoder features.”
k-means Mask Transformer (2022, methodology): “We adopt the same PQ-style loss, auxiliary semantic loss, mask-id cross-entropy loss, and pixel-wise instance discrimination loss [106].”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Recent work has proposed end-to-end approaches with Transformer based architectures [58, 17, 70, 16, 35, 68, 69, 33], for which the model directly predicts segmentation masks and optimizes based on a bipartite graph matching loss.”