Anchor DETR: Query Design for Transformer-Based Detector

By Yingming Wang, X. Zhang, Tong Yang, Jian‐Yuan Sun. 2022

603 citations recorded by the source corpus.

In this paper, we propose a novel query design for the transformer-based object detection. In previous transformer-based detectors, the object queries are a set of learned embeddings. However, each learned embedding does not have an explicit physical meaning and we cannot explain where it will focus on. It is difficult to optimize as the prediction slot of each object query does not have a specific mode. In other words, each object query will not focus on a specific region. To solve these problems, in our query design, object queries are based on anchor points, which are widely used in CNN-based detectors. So each object query focuses on the objects near the anchor point. Moreover, our query design can predict multiple objects at one position to solve the difficulty: ``one region, multiple objects''. In addition, we design an attention variant, which can reduce the memory cost while achieving similar or better performance than the standard attention in DETR. Thanks to the query design and the attention variant, the proposed detector that we called Anchor DETR, can achieve better performance and run faster than the DETR with 10x fewer training epochs. For example, it achieves 44.2 AP with 19 FPS on the MSCOCO dataset when using the ResNet50-DC5 feature for training 50 epochs. Extensive experiments on the MSCOCO benchmark prove the effectiveness of the proposed methods. Code is available at https://github.com/megvii-research/AnchorDETR.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “DETR was first proposed in [2] and then has been improved from many directions [4, 5, 12, 17, 33, 50, 64] in the past few years.”
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR (2022, methodology): “Anchor DETR (Wang et al., 2021) improves DETR by introducing 2D anchor points, which are updated layer by layer.”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “Further, detection queries (det queries) and segmentation queries (seg queries), which are initialized from learnable 3D anchor points [37] and fixed BEV points respectively, are fed into the transformer decoder and interact with multi-view image features.”
DETRs Beat YOLOs on Real-time Object Detection (2023, background): “Recently, the end-to-end Transformer-based detectors (DETRs) [4, 16, 22, 25, 33, 36, 41, 42] have received extensive attention from the academia due to their streamlined architecture and elimination of hand-crafted components.”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022, methodology): “To address such problems, many methods have been proposed, such as introducing deformable attention [41], decoupling positional and content information [25], providing spatial priors [11,39,37], etc.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Anchor DETR [176] proposes to use anchor points to replace the learnable query and also designs an efﬁcient self-attention head for faster training.”
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, background): “Analogy to anchor points in AnchorDETR[58], the output tokens from encoder are already learned well by directly aligning the SAM features, which can serve as anchor tokens for help masked tokens align via cross-attention decoder.”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, background): “Many follow-ups (Zhu et al. 2021; Meng et al. 2021; Gao et al. 2021; Dai et al. 2021; Wang et al. 2021; Liu et al. 2022; Li et al. 2022; Zhang et al. 2022) improve the model performance and training efﬁciency of DETR.”
YOLOv10: Real-Time End-to-End Object Detection (2024, background): “Since then, various DETR variants have been proposed to enhance its performance and efficiency [42, 67, 56, 30, 36, 28, 5, 77, 82].”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “Instead of performing object detection in the perspective view, DETR3D [59], PETR [30] and Graph-DETR3D [11] design DETR [74, 61]-based detection heads with learnable object queries in the 3D space.”
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment (2022, methodology): “All experiments adopt ResNet-50 [22] and do not use multiple patterns [57].”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, methodology): “[50,30,26] generate the object queries from anchor points or anchors that use position prior for fast convergence.”
A Survey of Visual Transformers (2021, methodology): “0 Anchor-DETR-DC5 [83] 50 151 39 14 7 44.”
FUTR3D: A Unified Sensor Fusion Framework for 3D Detection (2022, methodology): “Following Anchor-DETR [35], we first randomly initialize the queries with N q reference points { c i } N q i =1 , where c i ∈ [0 , 1] 3 presents relative coordinates in 3D space.”