Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

By Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, José Manuel Álvarez, Tong Lu, Ping Luo. 2021

191 citations recorded by the source corpus.

Panoptic segmentation involves a combination of joint semantic segmentation and instance segmentation, where image contents are divided into two types: things and stuff. We present Panoptic SegFormer, a general framework for panoptic segmentation with transformers. It contains three innovative components: an efficient deeply-supervised mask decoder, a query decoupling strategy, and an improved postprocessing method. We also use Deformable DETR to efficiently process multiscale features, which is a fast and efficient version of DETR. Specifically, we supervise the attention modules in the mask decoder in a layer-wise manner. This deep supervision strategy lets the attention modules quickly focus on meaningful semantic regions. It improves performance and reduces the number of required training epochs by half compared to Deformable DETR. Our query decoupling strategy decouples the responsibilities of the query set and avoids mutual interference between things and stuff. In addition, our post-processing strategy improves performance without additional costs by jointly considering classification and segmentation qualities to resolve conflicting mask overlaps. Our approach increases the accuracy 6.2% PQ over the baseline DETR model. Panoptic SegFormer achieves state-of-the-art results on COCO testdev with 56.2% PQ. It also shows stronger zero-shot robustness over existing methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Panoptic Segformer [31] separates foreground and background queries, but their background queries have fixed semantics, with each query corresponding to a pre-defined background category, limiting their ability to handle open-vocabulary categories.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “We list the most recent models for individual tasks, including Mask2Former [12], Panoptic SegFormer [47], KMaXDeepLab [93] for generic segmentation, and LAVT [86] for referring segmentation.”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, neutral): “0 — — — Panoptic SegFormer [33] Swin-L† [38] 221M 816G 353 24 55.”
DETRs with Hybrid Matching (2022, background): “The follow-up works have reformulated the object query to various variants for different visual recognition tasks such as mask query [9, 31], pose query [59], track query [49, 61], bins query [32], and so on.”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, methodology): “With the uniﬁed features generated by BEVFormer, the model can collaborate with different task-speciﬁc heads such as Deformable DETR [56] and mask decoder [22], for end-to-end 3D object detection and map segmentation.”
Vision Transformer Adapter for Dense Predictions (2022, background): “Take the image task as an instance, lacking prior information of images results in slower convergence and lower performance, and thus vanilla transformers are not competitive with dedicated transformers [45,75,78] for dense prediction tasks.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Recent work has proposed end-to-end approaches with Transformer based architectures [58, 17, 70, 16, 35, 68, 69, 33], for which the model directly predicts segmentation masks and optimizes based on a bipartite graph matching loss.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Inaddition to VIS, several works [163], [165], [225] have shown that query-based methods can naturally unify different segmentation tasks.”
Segment Everything Everywhere All at Once (2023, methodology): “Recently, the Detection Transformer (DETR)[31], a model based on the Transformer [44] architecture, has made signiﬁcant advances in segmentation [45, 6, 7, 46, 47] tasks.”
Planning-oriented Autonomous Driving (2022, methodology): “Following [56], we decompose the map query set into thing queries and stuff queries.”
PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (2023, background): “…detection (Liu et al., 2021; Wang et al., 2021; 2022; Ge et al., 2023; Carion et al., 2020), semantic segmentation (Zheng et al., 2021; Xie et al., 2021; Strudel et al., 2021) and so on (Sun et al., 2020; Li et al., 2022b; Zhao et al., 2021; Liu et al., 2022; He et al., 2022; Li et al., 2022a).”