Attention-Guided Unified Network for Panoptic Segmentation

By Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang. 2018

293 citations recorded by the source corpus.

This paper studies panoptic segmentation, a recently proposed task which segments foreground (FG) objects at the instance level as well as background (BG) contents at the semantic level. Existing methods mostly dealt with these two problems separately, but in this paper, we reveal the underlying relationship between them, in particular, FG objects provide complementary cues to assist BG understanding. Our approach, named the Attention-guided Unified Network (AUNet), is a unified framework with two branches for FG and BG segmentation simultaneously. Two sources of attentions are added to the BG branch, namely, RPN and FG segmentation mask to provide object-level and pixel-level attentions, respectively. Our approach is generalized to different backbones with consistent accuracy gain in both FG and BG segmentation, and also sets new state-of-the-arts both in the MS-COCO (46.5% PQ) and Cityscapes (59.0% PQ) benchmarks.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “TASCNet [48] and AUNet [51] propose a module to guide the fusion between ‘thing’ and ‘stuﬀ’ predictions, while Liu et al .”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Although we just use ResNet-101 as the backbone, we achieve slightly better results compared to the recent AUNet [20] which uses ResNeXt-152.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Then, the instance segments (‘thing’) and semantic segments (‘stuff’) [13] are fused by merging modules [54, 56, 74, 67, 101] to generate panoptic segmentation.”
Unifying Training and Inference for Panoptic Segmentation (2020, methodology): “A number of works have also attempted to encourage consistency between semantic and instance predictions by adding a communication mechanism between the two subnets [15, 17].”
k-means Mask Transformer (2022, methodology): “…box-based methods [54] that decompose panoptic segmentation into multiple surrogate tasks ( e.g ., predicting masks for each detected object bounding box [40], followed by fusing the instance segments (‘thing’) and semantic segments (‘stuﬀ’) [14] with merging modules [62,79,67,105,101,60]).”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “…of multiple stages involving object detection, segmentation, Figure 1: We formulate panoptic segmentation as a conditional discrete mask ( m ) generation problem for images (left) and videos (right), using a Bit Diffusion generative model [12]. merging multiple predictions [30, 34, 66, 40, 14].”
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers (2021, methodology): “Since then, models such as Panoptic FPN [7], UPSNet [9] and AUNet [20] have improved the accuracy and reduced the computational overhead by combining instance segmentation and semantic segmentation into a single model.”
EfficientPS: Efficient Panoptic Segmentation (2020, methodology): “Subsequently, Li et al (2019b) introduce Attentionguided Unified Network that uses proposal attention module and mask attention module for better segmentation of ‘stuff’ classes.”
Towards Bounding-Box Free Panoptic Segmentation (2020, methodology): “7 AUNet [22] 3 56.”
Panoptic SegFormer (2021, methodology): “Since then, models such as Panoptic FPN [2], UPSNet [4] and AUNet [23] have improved the accuracy and reduced the computational overhead by combining instance segmentation and semantic segmentation into a single model.”
Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation (2020, unknown): “Previous methods [28, 40, 23, 22] use complicated fusion modules to remap the outputs from two branches to a canvas which is of the same size as the semantic segmentation feature map to align foreground masks with background stuff, and thus introduces a considerable amount of parameters and high mem”