MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers

By Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan L. Yuille, Liang-Chieh Chen. 2020

612 citations recorded by the source corpus.

We present MaX-DeepLab, the first end-to-end model for panoptic segmentation. Our approach simplifies the current pipeline that depends heavily on surrogate sub-tasks and hand-designed components, such as box detection, non-maximum suppression, thing-stuff merging, etc. Although these sub-tasks are tackled by area experts, they fail to comprehensively solve the target task. By contrast, our MaX-DeepLab directly predicts class-labeled masks with a mask transformer, and is trained with a panoptic quality inspired loss via bipartite matching. Our mask transformer employs a dual-path architecture that introduces a global memory path in addition to a CNN path, allowing direct communication with any CNN layers. As a result, MaX-DeepLab shows a significant 7.1% PQ gain in the box-free regime on the challenging COCO dataset, closing the gap between box-based and box-free methods for the first time. A small variant of MaX-DeepLab improves 3.0% PQ over DETR with similar parameters and M-Adds. Furthermore, MaX-DeepLab, without test time augmentation, achieves new state-of-the-art 51.3% PQ on COCO test-dev set.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “Several other works use attention to process or refine latent spaces that interface with the input/output data using task- or domain-specific architectures [12, 45, 92].”
Perceiver: General Perception with Iterative Attention (2021, background): “…domain-agnostic analogue of models that stacks self-attention on top of convolutional feature maps to perform cheap but global processing on top or in conjunction with otherwise spatially localized convolutional feature maps (e.g. Carion et al. 2020; Locatello et al. 2020; Wang et al. 2021).”
Generalized Decoding for Pixel, Image, and Language (2022, background): “[73] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.”
Visual Prompt Tuning (2022, background): “The triumph of the Transformer architecture also extends to various computer vision tasks, including image classification [16,48], object detection [7,45], semantic segmentation [66,83], panoptic segmentation [73], video understanding [21] and fewshot learning [15], surpassing previous state-of-the-”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, background): “CMT-DeepLab‡ [59] MaX-S† [50] — — — 1025×2049 60k 64.”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, neutral): “Since deep learning, endto-end approaches proved their success on problems such as semantic segmentation [37,39], instance segmentation [22], panoptic segmentation [29, 52].”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “We ﬁnd panoptic architectures usually only report performance on a single panop-tic segmentation task [52], which does not guarantee good performance on other tasks (Figure 1).”
Per-Pixel Classification is Not All You Need for Semantic Segmentation (2021, methodology): “MaskFormer also outperforms recently proposed Max-DeepLab [42] without the need of special network design as well as sophisticated auxiliary losses (i.e., instance discrimination loss, mask-ID cross entropy loss, and per-pixel classification loss in [42]).”
K-Net: Towards Unified Image Segmentation (2021, methodology): “In contrast to K-Net that directly uses learned kernels to predict masks and progressively refines the masks and kernels, MaX-DeepLab [52] and MaskFormer [12] rely on queries and Transformer [51] to produce dynamic kernels for the final mask prediction.”
Segmenter: Transformer for Semantic Segmentation (2021, methodology): “The success of transformers in NLP has inspired several methods in computer vision combining CNNs with forms of selfattention to address object detection [6], semantic segmentation [49], panoptic segmentation [48], video processing [50] and few-shot classification [16].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video in”
ViViT: A Video Vision Transformer (2021, background): “ViT has inspired a large amount of follow-up work in the community, and we note that there are a number of concurrent approaches on extending it to other tasks in computer vision [71, 74, 84, 85] and improving its data-efﬁciency [64, 48].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “In addition to image classification, Transformers have also been applied to other compute vision tasks, including object detection [3, 54, 50, 10], segmentation [41, 44], image enhancement [4, 46], image generation [28, 6], video processing [48, 53], and vision-language tasks [26, 35, 7, 33, 21, 20,”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “Since its promise firstly demonstrated in Vision Transformer (ViT) [23], we have witnessed a flourish of full-Transformer models for image classification [57, 63, 67, 44, 80, 59], object detection [9, 91, 84, 20] and semantic segmentation [61, 65].”
A Survey on Vision Transformer (2020, methodology): “There have been a number of transformer models addressing these high-level vision tasks, such as DETR [14], deformable DETR [155] for object detection and Max-DeepLab [126] for segmentation.”
Large-scale pancreatic cancer detection via non-contrast CT and deep learning (2023, positive): “…diagnosis of pancreatic lesions if any abnormalities are detected in the second stage, integrated with an auxiliary memory transformer branch 24,25 to automatically encode the feature prototypes of the pancreatic lesions, such as local texture, position and pancreas shape, for more accurate…”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Recent work has proposed end-to-end approaches with Transformer based architectures [58, 17, 70, 16, 35, 68, 69, 33], for which the model directly predicts segmentation masks and optimizes based on a bipartite graph matching loss.”
Voxel Transformer for 3D Object Detection (2021, methodology): “MaX-DeepLab [31] utilizes a mask Transformer for panoptic segmentation.”
Medical Transformer: Gated Axial-Attention for Medical Image Segmentation (2021, methodology): “It was followed up by MaX-Deeplab [36] which uses a mask transformer to solve panoptic segmentation in an end-to-end way.”