Dynamic DETR: End-to-End Object Detection with Dynamic Attention

By Xiyang Dai, Yinpeng Chen, Jianwei Yang, Pengchuan Zhang, Lu Yuan, Lei Zhang. 2021

452 citations recorded by the source corpus.

In this paper, we present a novel Dynamic DETR (Detection with Transformers) approach by introducing dynamic attentions into both the encoder and decoder stages of DETR to break its two limitations on small feature resolution and slow training convergence. To address the first limitation, which is due to the quadratic computational complexity of the self-attention module in Transformer encoders, we propose a dynamic encoder to approximate the Transformer encoder’s attention mechanism using a convolution-based dynamic encoder with various attention types. Such an encoder can dynamically adjust attentions based on multiple factors such as scale importance, spatial importance, and representation (i.e., feature dimension) importance. To mitigate the second limitation of learning difficulty, we introduce a dynamic decoder by replacing the cross-attention module with a ROI-based dynamic attention in the Transformer decoder. Such a decoder effectively assists Transformers to focus on region of interests from a coarse-to-fine manner and dramatically lowers the learning difficulty, leading to a much faster convergence with fewer training epochs. We conduct a series of experiments to demonstrate our advantages. Our Dynamic DETR significantly reduces the training epochs (by 14×), yet results in a much better performance (by 3.6 on mAP). Meanwhile, in the standard 1× setup with ResNet-50 backbone, we archive a new state-of-the-art performance that further proves the learning effectiveness of the proposed approach.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Florence: A New Foundation Model for Computer Vision (2021, methodology): “We further compare Florence zero-shot with previous state-of-the-art detector4 (Dai et al., 2021a) (on COCO) fine-tunning on these tasks.”
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, methodology): “In high-level vision, Transformer has been widely applied in image classification [7, 19, 25, 27, 49, 65], object detection [6, 22, 23, 60, 86, 93], semantic segmentation [16, 69,76,77,90,91], human pose estimation [14,15,32,37,42, 43, 53, 79, 89], etc.”
DETRs with Hybrid Matching (2022, methodology): “With the pioneering work DETR [5] introducing transformers [63] to 2D object detection, more and more follow-up works [11, 15, 50, 68] have built various advanced extensions based on DETR because it removes the need for many hand-designed components like non-maximum suppression [53] or initial ancho”
ActionFormer: Localizing Moments of Actions with Transformers (2022, background): “Transformer have also been explored in object detection [11, 17, 69, 87], semantic segmentation [15, 71, 73], and more recently video representation learning [3,23,49].”
Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement (2023, background): “…in many computer vision tasks and achieved impressive results in high-level vision ( e.g. , image classification [2, 4, 14], semantic segmentation [7, 55, 67], object detection [3, 13, 62], etc. ) and low-level vision ( e.g. , image restoration [6, 11, 60], image synthesis [20, 21, 64], etc. ).”
Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction (2021, background): “cess in many high-level vision tasks, such as image classification [2, 15, 17, 31], object detection [1, 13, 60, 64], segmentation [8, 55, 63], human pose estimation [7, 25, 26, 56], etc.”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, background): “Many follow-ups (Zhu et al. 2021; Meng et al. 2021; Gao et al. 2021; Dai et al. 2021; Wang et al. 2021; Liu et al. 2022; Li et al. 2022; Zhang et al. 2022) improve the model performance and training efﬁciency of DETR.”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “From the feature re-weighting view, Dynamic-DETR [234] introduces dynamic attention to both the encoder and decoder parts of DETR using RoI-wise dynamic convolution.”
DN-DETR: Accelerate DETR Training by Introducing Query DeNoising (2022, methodology): “Note that Dynamic DETR [6] adopts a dynamic encoder, for a fair comparison, we also compare with its version without a dynamic encoder.”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022, methodology): “The best detection models nowadays are based on improved classical detectors like DyHead [8] and HTC [4].”
A Survey of Visual Transformers (2021, methodology): “Thus, the decoder is necessary but it requires more spatial prior [77], [81]– [84], [86], [87] owing to its slow convergence [88].”