Rethinking Transformer-based Set Prediction for Object Detection

By Zhiqing Sun, Shengcao Cao, Yiming Yang, Kris M. Kitani. 2020

383 citations recorded by the source corpus.

DETR is a recently proposed Transformer-based method which views object detection as a set prediction problem and achieves state-of-the-art performance but demands extra-long training time to converge. In this paper, we investigate the causes of the optimization difficulty in the training of DETR. Our examinations reveal several factors contributing to the slow convergence of DETR, primarily the issues with the Hungarian loss and the Transformer cross-attention mechanism. To overcome these issues we propose two solutions, namely, TSP-FCOS (Transformer-based Set Prediction with FCOS) and TSP-RCNN (Transformer-based Set Prediction with RCNN). Experimental results show that the proposed methods not only converge much faster than the original DETR, but also significantly outperform DETR and other baselines in terms of detection accuracy. Code is released at https://github.com/Edward-Sun/TSP-Detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Masked-attention Mask Transformer for Universal Image Segmentation (2021, background): “However, recent studies [22,46] suggest that the slow convergence of Transformer-based models is due to global context in the cross-attention layer, as it takes many training epochs for cross-attention to learn to attend to localized object regions [46].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR (2022, methodology): “We compare our proposed DABDETR with DETR (Carion et al., 2020), Faster RCNN (Ren et al., 2017), Anchor DETR (Wang et al., 2021), SMCA (Gao et al., 2021), Deformable DETR (Zhu et al., 2021), TSP (Sun et al., 2020), and Conditional DETR (Meng et al., 2021).”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, background): “[44] attributes the slow convergence to the cross attention mechanism and designs a encoder-only DETR.”
A Survey on Vision Transformer (2020, methodology): “[117] investigate the slow convergence problem of DETR model and reveal that the cross-attention module in the transformer decoder is the main reason behind.”
Auto-scaling Vision Transformers without Training (2022, background): “Recent works explored transformers to vision problems: image classiﬁcation (Dosovitskiy et al., 2020), object detection (Carion et al., 2020; Zhu et al., 2020; Zheng et al., 2020; Dai et al., 2020; Sun et al., 2020), segmentation (Chen et al., 2020; Wang et al., 2020), etc.”
Referring Transformer: A One-step Approach to Multi-task Visual Grounding (2021, background): “Authors of [47] show that a possible reason could lie in the Hungarian matching loss used to match sets of predictions.”
Fast Convergence of DETR with Spatially Modulated Co-Attention (2021, methodology): “TSP-FCOS and TSP-RCNN [36] combines DETR’s Hungarian matching with FCOS [37] and RCNN [33] detectors, which results in faster convergence and better performance than DETR.”
Visual Grounding with Transformers (2021, background): “Encouraged by its powerful representation ability, researchers attempt to extend this architecture for vision tasks such as object detection [3, 53, 35, 8], segmentation [39, 27], lane detection [20] and others [37].”
Continual Detection Transformer for Incremental Object Detection (2023, methodology): “State-of-the-art methods like DETR [5,10,29,49,58,61] build on powerful visual transformers to solve the object detection problem.”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection (2022, methodology): “[34] designed an encoder-only DETR without using a decoder.”
A Survey of Visual Transformers (2021, methodology): “Thus, the decoder is necessary but requires few stacks [71] owing to its slow convergence [72].”
All Tokens Matter: Token Labeling for Training Better Vision Transformers (2021, background): “Due to their strong capability of capturing spatial information, transformers have also been successfully applied to a variety of vision problems, including low-level vision tasks like image enhancement [7, 45], as well as more challenging tasks such as image classification [9, 15], object detection”