Deformable DETR: Deformable Transformers for End-to-End Object Detection

By Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai. 2020

7,924 citations recorded by the source corpus.

DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10$\times$ less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach. Code shall be released.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, background): “including image classification [2], [3], [4], object detection [5], [6] and image processing like denoising, super-resolution and deraining [7].”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “More recently, the encoder-decoder design in Transformer has been applied for the object detection and instance segmentation tasks [8, 13, 76, 52].”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Recently, Detection Transformer (DETR) [1] that is based on Transformer [44] has achieved significant progress in many detection [28, 34, 37, 55, 60] and segmentation models [5, 18, 29, 32].”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, methodology): “We propose Grounding DINO, which extends a closed-set detector DINO by performing vision-language modality fusion at multiple phases, including a feature enhancer, a language-guided query selection module, and a cross-modality decoder.”
Generalized Decoding for Pixel, Image, and Language (2022, background): “” means multi-scale deformable attention [99] that is used in Mask2Former [12].”
Grounded Language-Image Pre-training (2021, background): “…T ∈ { 0 , 1 } N × c is the target matching between regions and classes computed from the classical many-to-1 matching [9,36,48,49] or the bipartite Hungarian match [4,10,76]. loss ( S ; T ) is typically a cross-entropy loss for two-stage detectors and a focal loss [36] for one-stage detectors.”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, methodology): “While pursuing detection heads with fewer inductive biases is an active area of work, leading methods like DETR [5] are challenging to train and still benefit from detection-specific prior knowledge [58].”
Instance Segmentation for Autonomous Log Grasping in Forestry Operations (2022, methodology): “A six-layer multi-scale deformable attention Transformer (MSDeformAttn) [31] then decodes the feature maps to generate high-resolution per-pixel embeddings.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “For each resolution, we add both a sinusoidal positional embedding epos 2 RHlWl⇥C , following [5], and a learnable scale-level embedding elvl 2 R1⇥C , following [66].”
K-Net: Towards Unified Image Segmentation (2021, methodology): “For panoptic and instance segmentation, we use λcls = 2 for Focal loss following previous methods [72], and empirically find λseg = 1, λce = 1, λdice = 4 work best.”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, methodology): “Based on DETR, deformable DETR [82] further adopts a deformable attention layer to focus on a sparse set of contextual elements, obtaining faster convergence and better performance.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, background): “Both classes of methods have boosted various vision tasks such as image classification [18, 5, 72, 37, 57, 89, 28], object detection [91, 80, 72, 36, 10, 108], semantic segmentation [15, 106, 39, 29, 109, 107], video recognition [91, 18], image generation [73, 35], and panoptic segmentation [89].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Recently, transformer architectures [7, 26, 86, 97, 104, 109] shine in semantic segmentation, focusing on multi-scale feature fusion [12, 84, 89, 101] and contextual feature aggregation [58, 79].”
Transformers in Vision: A Survey (2021, methodology): “D-DETR [14] ICLR’21 AP COCO 43.8 a) Achieves better performance on small objects than DETR [13], b) Faster convergence than DETR [13] Obtain SOTA results with 52.3 AP but with two stage detector design and test time augmentations.”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, methodology): “Recent works have explored applying transformers to various vision tasks: image classiﬁcation [5, 12], object detection [3, 61, 58, 8, 34], segmentation [4, 40], image enhancement [4, 45], image generation [27], video processing [60, 53], and 3D point cloud processing [56].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
Transformer in Transformer (2021, background): “Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
Dynamic Head: Unifying Object Detection Heads with Attentions (2021, methodology): “We compare our methods with state-of-the-art detectors [35, 31, 21, 4, 2, 27, 8], including some concurrent works [38, 5].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “In addition to image classification, Transformers have also been applied to other computer vision tasks, including object detection [4, 58, 54, 11], segmentation [45, 48], image enhancement [5, 50], image generation [30, 7], video processing [52, 57], and vision-language tasks [28, 38, 8, 36, 22, 21”