DN-DETR: Accelerate DETR Training by Introducing Query DeNoising

By Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel Ming-shuan Ni, Lei Zhang. 2022

1,199 citations recorded by the source corpus.

We present in this paper a novel denoising training method to speedup DETR (DEtection TRansformer) training and offer a deepened understanding of the slow convergence issue of DETR-like methods. We show that the slow convergence results from the instability of bipartite graph matching which causes inconsistent optimization goals in early training stages. To address this issue, except for the Hungarian loss, our method additionally feeds ground-truth bounding boxes with noises into Transformer decoder and trains the model to reconstruct the original boxes, which effectively reduces the bipartite graph matching difficulty and leads to a faster convergence. Our method is universal and can be easily plugged into any DETR-like methods by adding dozens of lines of code to achieve a remarkable improvement. As a result, our DN-DETR results in a remarkable improvement (+1.9AP) under the same setting and achieves the best result (AP 43.4 and 48.6 with 12 and 50 epochs of training respectively) among DETR-like methods with ResNet-50 backbone. Compared with the baseline under the same setting, DN-DETR achieves comparable performance with 50% training epochs. Code is available at https://github.com/FengLi-ust/DN-DETR.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Model Architectures

Citations in the corpus

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, methodology): “DN-DETR [24] proposes a query denoising approach to stabilizing the bipartite matching.”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, methodology): “Architectures based on the transformer encoder-decoder structure [3,31,36,64] have proved effective in object detection since the introduction of DETR [3].”
A Survey of Visual Transformers (2021, methodology): “[92] exploit a denoising DETR (DN-DETR) to mitigate the instability of bipartite matching.”
Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, neutral): “However, the mask classification paradigm was originated from query-based detectors, such as DETR [2] and its variants [24, 59, 66], where we train the model to predict proposals and class labels with learnable queries.”
YOLOv10: Real-Time End-to-End Object Detection (2024, background): “Since then, various DETR variants have been proposed to enhance its performance and efficiency [42, 67, 56, 30, 36, 28, 5, 77, 82].”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, methodology): “We extend query denoise of DN-DETR [18] to accelerate convergence of 3D object detection.”
Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection (2023, methodology): “The query denoising [22] is also conducted following PETRv2 [31].”
DETRs Beat YOLOs on Real-time Object Detection (2023, background): “Recently, the end-to-end Transformer-based detectors (DETRs) [4, 16, 22, 25, 33, 36, 41, 42] have received extensive attention from the academia due to their streamlined architecture and elimination of hand-crafted components.”
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment (2022, methodology): “To make direct comparisons with DN-DETR [13], we report the best results across different numbers of denoising queries in Figure 10 ( 38 .”
DETRs with Hybrid Matching (2022, methodology): “Motivated by the success of DETR on a wide variety of vision tasks, many follow-up efforts have improved DETR from various aspects, including redesigning more advanced transformer encoder [81, 13, 14] or transformer decoder architectures [44, 73, 81, 4] or query formulations [62, 34, 22, 74].”
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…advancements have been made across a variety of tasks, including image recognition [47, 31, 18, 83, 17], generic object detection [49, 87, 43, 27, 77, 36, 19, 38, 51, 50, 20, 30], generic image segmentation [9, 8, 26, 29, 78, 88, 79, 25, 79, 16, 28], referring object detection and…”
YOLOv12: Attention-Centric Real-Time Object Detectors (2025, neutral): “Recently, an end-to-end object detection method, namely RT-DETR [66], improved traditional end-to-end detectors [7, 33, 37, 42, 71] to meet real-time requirements by designing an efficient encoder and an uncertainty-minimal query selection mechanism.”
SDDGR: Stable Diffusion-Based Deep Generative Replay for Class Incremental Object Detection (2024, methodology): “Furthermore, as the Transformer-based architectures [3, 29, 71] are introduced as the alternative to the CNN-based approaches [31, 48], the CIOD for Transformer-based object detector is also proposed.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, background): “It improves interpretability and accelerates training c vergence (Li et al., 2022).”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, methodology): “Many follow-ups (Zhu et al. 2021; Meng et al. 2021; Gao et al. 2021; Dai et al. 2021; Wang et al. 2021; Liu et al. 2022; Li et al. 2022; Zhang et al. 2022) improve the model performance and training efﬁciency of DETR.”
DETRs with Collaborative Hybrid Assignments Training (2022, methodology): “The instability (IS) [14] of Deformable-DETR and CoDeformable-DETR on COCO dataset.”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022, methodology): “Recently, DNDETR [17] introduces a denoising training method to speed up DETR training.”