Conditional DETR for Fast Training Convergence

By Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, Jingdong Wang. 2021

1,038 citations recorded by the source corpus.

The recently-developed DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings for localizing the four extremities and predicting the box, which increases the need for high-quality content embeddings and thus the training difficulty.Our approach, named conditional DETR, learns a conditional spatial query from the decoder embedding for decoder multi-head cross-attention. The benefit is that through the conditional spatial query, each cross-attention head is able to attend to a band containing a distinct region, e.g., one object extremity or a region inside the object box. This narrows down the spatial range for localizing the distinct regions for object classification and box regression, thus relaxing the dependence on the content embeddings and easing the training. Empirical results show that conditional DETR converges 6.7× faster for the backbones R50 and R101 and 10× faster for stronger backbones DC5-R50 and DC5-R101. Code is available at https://github.com/Atten4Vis/ConditionalDETR.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Recently, Detection Transformer (DETR) [1] that is based on Transformer [43] has achieved significant progress in many detection [27, 33, 36, 54, 59] and segmentation models [5, 18, 28, 31].”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, methodology): “Each decoder query contains two parts: content part and positional part [33], respectively.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, background): “Concurrent work has proposed other variants of cross-attention [22, 40] that aim to improve the convergence and performance of DETR [5] for object detection.”
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR (2022, methodology): “Related discussion can also be found in Conditional DETR (Meng et al., 2021).”
DN-DETR: Accelerate DETR Training by Introducing Query DeNoising (2022, methodology): “More recent works propose to associate each DETR query with a specific spatial position rather than multiple positions for more efficient feature probing [11, 12, 18, 20].”
Motion Transformer with Global Intention Localization and Local Movement Refinement (2022, methodology): “Our approach is inspired by DETR [4] and its follow-up works [63, 32, 56, 25, 28, 10, 59], especially DAB-DETR [28], where the object query is considered as the positional embedding of a spatial anchor box.”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022, methodology): “To address such problems, many methods have been proposed, such as introducing deformable attention [41], decoupling positional and content information [25], providing spatial priors [11,39,37], etc.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “• Adding Position Information into Query: Conditional DETR [174] ﬁnds cross-attention in DETR relies highly on the content embeddings for localizing the four extremities.”
Anchor DETR: Query Design for Transformer-Based Detector (2022, methodology): “Concurrent with us, the Conditional DETR (Meng et al. 2021) encodes the reference point as the query position embedding.”
Open-Vocabulary DETR with Conditional Matching (2022, background): “Although DETR [2] and its follow-ups [39, 23] are developed for the object detection task, they can also be extended to the instance segmentation task.”
DETRs with Collaborative Hybrid Assignments Training (2022, methodology): “Method Backbone Multi-scale #query #epochs AP AP50 AP75 APS APM APL Conditional-DETR [23] R50 7 300 108 43.”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, methodology): “We use the term “query” as the input of the Transformer decoder layers in this paper, following the common practice in the Transformer and DETR-related literature (Vaswani et al. 2017; Meng et al. 2021; Liu et al. 2022).”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…advancements have been made across a variety of tasks, including image recognition [47, 31, 18, 83, 17], generic object detection [49, 87, 43, 27, 77, 36, 19, 38, 51, 50, 20, 30], generic image segmentation [9, 8, 26, 29, 78, 88, 79, 25, 79, 16, 28], referring object detection and…”
End-to-End Instance Edge Detection (2022, methodology): “Most of the hyperparameters follow the implementation in [48].”
YOLOv10: Real-Time End-to-End Object Detection (2024, background): “Since then, various DETR variants have been proposed to enhance its performance and efficiency [42, 67, 56, 30, 36, 28, 5, 77, 82].”
DiffusionDet: Diffusion Model for Object Detection (2022, background): “Since then, the query-based detection paradigm has attracted great attention and inspired a series of following works [12,24,43,51,58,64,66,89,90,107,110, 114].”
Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation (2022, methodology): “For example, Mask2Former uses such decoder queries to dot-product First, its queries follow the design in DETR [1] without being able to utilize better positional priors as studied in Conditional DETR [26], Anchor DETR [34], and DAB-DETR [22].”