K-Net: Towards Unified Image Segmentation

By Wenwei Zhang, Jiangmiao Pang, Kai Chen, Chen Change Loy. 2021

480 citations recorded by the source corpus.

Semantic, instance, and panoptic segmentations have been addressed using different and specialized frameworks despite their underlying connections. This paper presents a unified, simple, and effective framework for these essentially similar tasks. The framework, named K-Net, segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class. To remedy the difficulties of distinguishing various instances, we propose a kernel update strategy that enables each kernel dynamic and conditional on its meaningful group in the input image. K-Net can be trained in an end-to-end manner with bipartite matching, and its training and inference are naturally NMS-free and box-free. Without bells and whistles, K-Net surpasses all previous published state-of-the-art single-model results of panoptic segmentation on MS COCO test-dev split and semantic segmentation on ADE20K val split with 55.2% PQ and 54.3% mIoU, respectively. Its instance segmentation performance is also on par with Cascade Mask R-CNN on MS COCO with 60%-90% faster inference speeds. Code and models will be released at https://github.com/ZwwWayne/K-Net/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Segment Anything (2023, background): “, joint semantic, instance, and panoptic segmentation [114, 19, 54], but the training and test tasks are the same.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, background): “K-Net [62] further extends set prediction to instance segmentation.”
Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation (2022, methodology): “The ambition to bridge different tasks gives rise to more advanced methods like HTC [2] for object detection and instance segmentation and Panoptic FPN [16], K-net [38] for instance, panoptic, and semantic segmentation.”
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers (2021, methodology): “6% PQ on COCO val with ResNet-50 as the backbone and singlescale input, and it surpasses previous methods K-Net [4] and DETR [1] over 2.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Recent work has proposed end-to-end approaches with Transformer based architectures [58, 17, 70, 16, 35, 68, 69, 33], for which the model directly predicts segmentation masks and optimizes based on a bipartite graph matching loss.”
ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (2024, methodology): “Following this, some methods [27, 43, 48] introduced significant improvements by employing dynamic convolutions in a fully convolutional paradigm.”
Panoptic SegFormer (2021, background): “For example, several works [11–14] achieve this with fully convolutional frameworks.”
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, methodology): “We observe that with a similar performance to K-Net [11] in RQ, our best model performs better in SQ.”
OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping (2022, methodology): “(2) UPerNet with Swin-B and Twins, as well as SegFormer and K-Net perform better than DeepLabV3 and HRNet.”
LISA: Reasoning Segmentation via Large Language Model (2023, background): “Research on instance segmentation (He et al., 2017; Zhang et al., 2021; Cheng et al., 2022) and panoptic segmentation (Kirillov et al.”
k-means Mask Transformer (2022, methodology): “K-Net [109] proposed dynamic kernels for generating the masks.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Inaddition to VIS, several works [163], [165], [225] have shown that query-based methods can naturally unify different segmentation tasks.”