Fast Convergence of DETR with Spatially Modulated Co-Attention

By Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, Hongsheng Li. 2021

404 citations recorded by the source corpus.

The recently proposed Detection Transformer (DETR) model successfully applies Transformer to objects detection and achieves comparable performance with two-stage object detection frameworks, such as Faster-RCNN. However, DETR suffers from its slow convergence. Training DETR [4] from scratch needs 500 epochs to achieve a high accuracy. To accelerate its convergence, we propose a simple yet effective scheme for improving the DETR framework, namely Spatially Modulated Co-Attention (SMCA) mechanism. The core idea of SMCA is to conduct location-aware co-attention in DETR by constraining co-attention responses to be high near initially estimated bounding box locations. Our proposed SMCA increases DETR’s convergence speed by replacing the original co-attention mechanism in the decoder while keeping other operations in DETR unchanged. Furthermore, by integrating multi-head and scale-selection attention designs into SMCA, our fully-fledged SMCA can achieve better performance compared to DETR with a dilated convolution-based backbone (45.6 mAP at 108 epochs vs. 43.3 mAP at 500 epochs). We perform extensive ablation studies on COCO dataset to validate SMCA. Code is released at https://github.com/gaopengcuhk/SMCA-DETR.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “DETR was first proposed in [2] and then has been improved from many directions [4, 5, 12, 17, 33, 50, 64] in the past few years.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, background): “Concurrent work has proposed other variants of cross-attention [22, 40] that aim to improve the convergence and performance of DETR [5] for object detection.”
CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “Visual understanding tasks, such as classification [12, 15, 20, 25, 32, 46, 60], object detection [5, 16, 52], and semantic segmentation [42], have been improved significantly based on the better architecture designs and large-scale high-quality datasets.”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, background): “The accuracy of vision tasks, such as classification [14, 24, 28, 37, 45, 61], detection [5, 19, 53] and segmentation [69], has been boosted”
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR (2022, methodology): “We compare our proposed DABDETR with DETR (Carion et al., 2020), Faster RCNN (Ren et al., 2017), Anchor DETR (Wang et al., 2021), SMCA (Gao et al., 2021), Deformable DETR (Zhu et al., 2021), TSP (Sun et al., 2020), and Conditional DETR (Meng et al., 2021).”
Instances as Queries (2021, methodology): “Follow-up works [63, 44, 45, 19, 60, 16] in object detection improve this query based approach and achieve comparable performance with state-of-the-art detectors such as Cascade RCNN [4].”
DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding (2022, background): “Many follow-ups (Zhu et al. 2021; Meng et al. 2021; Gao et al. 2021; Dai et al. 2021; Wang et al. 2021; Liu et al. 2022; Li et al. 2022; Zhang et al. 2022) improve the model performance and training efﬁciency of DETR.”
Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training (2022, methodology): “Motivated by this, Vision Transformers [14] and DETR [6] introduce the transformer architecture into computer vision, and stimulate follow-up works to effectively apply transformers to a wide range of vision tasks, such as image classification [43, 30, 31], object detection [66, 64, 16], semantic se”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022, methodology): “To address such problems, many methods have been proposed, such as introducing deformable attention [41], decoupling positional and content information [25], providing spatial priors [11,39,37], etc.”
k-means Mask Transformer (2022, background): “Each object query needs to learn to highlight the most distinguishable features among the abundant pixels in the cross-attention learning process, which subsequently leads to slow training convergence and thus inferior performance [116,37].”
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment (2022, methodology): “SMCA [16] uses the Gaussian-like weight for spatial modulation.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “SMAC [231] conducts location-aware co-attention by sampling features of high near estimated bounding box locations.”
RomeBERT: Robust Training of Multi-Exit BERT (2021, methodology): “C L ] 24 J a n 2021 computation efﬁciency by means of feature clustering (Vyas et al., 2020; Zheng et al., 2020; Ki-taev et al., 2020), low rank approximation (Choro-manski et al., 2020), attention sparse regularization (Gao et al., 2021), and reversible architecture (Kitaev et al., 2020).”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, background): “SMAC [14] predicts 2D Gaussian-like weight map as spatial prior for each query.”
End-to-End Instance Edge Detection (2022, background): “As such, many follow-up works to DETR aim at improving its convergence speed via additional priors [69,17,48] or by reducing transformer density [69].”
A Survey on Vision Transformer (2020, background): “[71] proposed the Spatially Modulated Co-Attention (SMCA) mechanism to accelerate the convergence by constraining co-attention responses to be high near initially estimated bounding box locations.”