Instances as Queries

By Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, Wenyu Liu. 2021

347 citations recorded by the source corpus.

We present QueryInst, a new perspective for instance segmentation. QueryInst is a multi-stage end-to-end system that treats instances of interest as learnable queries, enabling query based object detectors, e.g., Sparse RCNN, to have strong instance segmentation performance. The attributes of instances such as categories, bounding boxes, instance masks, and instance association embeddings are represented by queries in a unified manner. In QueryInst, a query is shared by both detection and segmentation via dynamic convolutions and driven by parallellysupervised multi-stage learning. We conduct extensive experiments on three challenging benchmarks, i.e., COCO, CityScapes, and YouTube-VIS to evaluate the effectiveness of QueryInst in object detection, instance segmentation, and video instance segmentation tasks. For the first time, we demonstrate that a simple end-to-end query based framework can achieve the state-of-the-art performance in various instance-level recognition tasks. Code is available at https://github.com/hustvl/QueryInst.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Instance Segmentation for Autonomous Log Grasping in Forestry Operations (2022, methodology): “QueryInst [19] further adapted DETR to instance segmentation, by adding dynamic mask heads.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, background): “However, recent work still focuses on advancing specialized architectures [20, 39, 45], which raises the question: why haven’t universal architectures replaced specialized ones? Although existing universal architectures are flexible enough to tackle any segmentation task, as shown in Figure 1, in pr”
SOLQ: Segmenting Objects by Learning Queries (2021, methodology): “‘ † ’ and ‘*’ are the results reported in ISTR [24] and QueryInst [24], respectively.”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, methodology): “We also evaluate nine open-source state-of-the-art methods (including FEELVOS [32], IoUTracker+ [39], MaskTrack R-CNN [39], SipMask [8], STEm-Seg [2], STMask [22], TraDeS [37], CrossVIS [40], and QueryInst [12]) on OVIS to analyze the OVIS dataset and serve as baselines for the OVIS challenge and fu”
Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation (2022, methodology): “6 441 80M 5 QueryInst [10] 36 300 queries 40.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “QueryInst [159] adds mask heads and reﬁnes mask query with dynamic convolution.”
DETRs with Hybrid Matching (2022, background): “Since the success of pioneering work DEtection TRansformer (DETR) [5] on object detection tasks, DETR-based approaches have achieved significant progress on various fundamental vision recognition tasks such as object detection [74, 81, 44, 52], instance segmentation [23, 12, 11, 70], panoptic segmen”
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers (2021, methodology): “In addition, our approach also achieves competitive performance with current methods [13, 14] on the instance segmentation task.”
On Vision Features in Multimodal Machine Translation (2022, methodology): “, 2020) and QueryInst (Fang et al., 2021) for their strong performance.”
Learning to Upsample by Learning to Sample (2023, background): “Dense prediction refers to a branch of tasks that require point-wise label prediction, such as semantic/instance/panoptic segmentation [2, 39, 40, 8, 7, 13, 11, 16, 19], object detection [33, 4, 24, 36], and monocular depth estimation [38, 18, 3, 21].”
What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2022, methodology): “We also evaluate instance segmentation models trained explicitly on COCO, like Mask RCNN [15] with a ResNet-101 backbone [16], QueryInst [17] with ResNet-101-FPN [18], and Mask2Former [19] with Swin-S [20], all implemented in the MMDetection library [21].”
A Survey of Visual Transformers (2021, methodology): “QueryInst [101] attains the SoTA among various Trans-former models.”