Masked-attention Mask Transformer for Universal Image Segmentation

By Bowen Cheng, Ishan Misra, Alexander Gerhard Schwing, Alexander Kirillov, Rohit Girdhar. 2021

4,195 citations recorded by the source corpus.

Image segmentation groups pixels with different semantics, e.g., category or instance membership. Each choice of semantics defines a task. While only the semantics of each task differ, current research focuses on designing spe-cialized architectures for each task. We present Masked- attention Mask Transformer (Mask2Former), a new archi-tecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components in-clude masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most no-tably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU onADE20K).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Quoc V. Le
4. Dacheng Tao
5. Luc Van Gool
6. Zhe Gan
7. Jiashi Feng
8. Alan L. Yuille

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Xiaogang Wang
5. Trevor Darrell
6. Jitendra Malik
7. Alan L. Yuille
8. Chunhua Shen

In Model Architectures

1. Jianfeng Gao
2. Jianwei Yang
3. Lu Yuan
4. Alan L. Yuille
5. Jiashi Feng
6. Bin Xiao
7. Xiaogang Wang
8. Hongsheng Li

Citations in the corpus

• Segment Anything (2023, background): “In a multi-task system, a single model performs a fixed set of tasks, e.g ., joint semantic, instance, and panoptic segmentation [114, 19, 54], but the training and test tasks are the same.”
• A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Mask2Former [5] and MaskDINO [28] treat foreground and background equally when conducting panoptic segmentation, resulting in suboptimal mask average precision (AP) for foreground objects compared to the same model solely trained on foreground classes (instance segmentation).”
• Generalized Decoding for Pixel, Image, and Language (2022, methodology): “(2) reduces to Mask2former [12], but with open-vocabulary capacity since we use mask-text matching for mask classification.”
• OneFormer: One Transformer to Rule Universal Image Segmentation (2022, methodology): “(ii) How can the multi-task model better learn inter-task and inter-class differences during the single joint training process? Following the recent success of transformer frameworks [3,12,21,22,27,38,60] in computer vision, we formulate our framework as a transformer-based approach, which can be gu”
• Recent Advances of Continual Learning in Computer Vision: An Overview (2021, neutral): “To solve this problem, they proposed to separate parameters into global and task-specific parameters, whereby each client can obtain a weighted combination of the task-specific parameters of the other clients for knowledge transfer.”
• Segment Everything Everywhere All at Once (2023, methodology): “We first pool the corresponding region from the image feature through point sampling [6].”
• Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, positive): “Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. menters [9, 10, 47, 55].”
• DN-DETR: Accelerate DETR Training by Introducing Query DeNoising (2022, background): “We also show the feasibility of adding denoising training to segmentation models such as Mask2Former [5].”
• EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, methodology): “We follow the task transfer pipelines of ViT-Adapter [23]+mask2former [24] but with a weakened model adaptation processes due to GPU memory limitation (40GB of VRAM): (i) relative position biases [86] are not applied.”
• Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation (2022, methodology): “As an evidence, our FPS is much higher than Mask2Former [4] (14.”
• LISA: Reasoning Segmentation via Large Language Model (2023, background): “Research on instance segmentation (He et al., 2017; Zhang et al., 2021; Cheng et al., 2022) and panoptic segmentation (Kirillov et al.”
• Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, methodology): “We use Mask2Former [12] as our semantic segmentation model.”
• Open-Vocabulary Panoptic Segmentation with MaskCLIP (2022, methodology): “The class-agnostic mask proposal networks we use in those methods are trained using Mask2Former other than noted.”
• MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation (2023, methodology): “We evaluate the results with an off-the-shelf segmentation model (Cheng et al., 2022) on the generated images, and measure the Intersection over Union (IoU) w.”