Efficient DETR: Improving End-to-End Object Detector with Dense Prior

By Z. Yao, Jiangbo Ai, Boxun Li, Chi Zhang. arXiv.org, 2021

356 citations recorded by the source corpus.

The recently proposed end-to-end transformer detectors, such as DETR and Deformable DETR, have a cascade structure of stacking 6 decoder layers to update object queries iteratively, without which their performance degrades seriously. In this paper, we investigate that the random initialization of object containers, which include object queries and reference points, is mainly responsible for the requirement of multiple iterations. Based on our findings, we propose Efficient DETR, a simple and efficient pipeline for end-to-end object detection. By taking advantage of both dense detection and sparse set detection, Efficient DETR leverages dense prior to initialize the object containers and brings the gap of the 1-decoder structure and 6-decoder structure. Experiments conducted on MS COCO show that our method, with only 3 encoder layers and 1 decoder layer, achieves competitive performance with state-of-the-art object detection methods. Efficient DETR is also robust in crowded scenes. It outperforms modern detectors on CrowdHuman dataset by a large margin.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/37b66aefc502b205da52a0a811fd9a3bafb399f3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Open-Vocabulary Object Detection (2023, methodology): “We introduce an objectness head which predicts the likelihood that an output token actually represents an object, and compute boxes, class scores, and losses only for the top k tokens by objectness, similar to Efficient DETR [34].”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022, methodology): “Similarly, Efficient DETR [39] also selects top K features based on the objectiveness (class) score of each encoder feature.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “The Efﬁcient DETR [241] adopts hybrid designs by including query-based and dense anchor-based predictions in one framework.”
DN-DETR: Accelerate DETR Training by Introducing Query DeNoising (2022, methodology): “[19] utilized a Region Proposal Network (RPN) to propose top-K anchor points.”
Simple Open-Vocabulary Object Detection with Vision Transformers (2022, methodology): “We would like to thank Sunayana Rane and Rianne van den Berg for help with the DETR implementation, Lucas Beyer for the data deduplication code, and Yi Tay for useful advice.”
A Survey on Vision Transformer (2020, background): “[257] pointed out that the random initialization in DETR is the main reason for the requirement of multiple decoder layers and slow convergence.”
TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (2022, methodology): “Recently, it has been observed in 2D object detection [56] that with a better initialization of object queries, the gap between 1-layer structure and 6-layer structure could be bridged.”
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR (2022, background): “Efficient DETR introduces a dense prediction module to select top-K object queries (Yao et al., 2021) and Anchor DETR formulates queries as 2D anchor points (Wang et al.”
DETRs Beat YOLOs on Real-time Object Detection (2023, methodology): “And for the latter, previous works [39, 41, 42] show that the hard-to-optimize object queries hinder the performance of DETRs and propose the query selection schemes to replace the vanilla learnable embeddings with encoder features.”