UP-DETR: Unsupervised Pre-training for Object Detection with Transformers

By Zhigang Dai, Bolun Cai, Yugeng Lin, Junying Chen. 2020

571 citations recorded by the source corpus.

Object detection with transformers (DETR) reaches competitive performance with Faster R-CNN via a transformer encoder-decoder architecture. Inspired by the great success of pre-training transformers in natural language processing, we propose a pretext task named random query patch detection to Unsupervisedly Pre-train DETR (UP-DETR) for object detection. Specifically, we randomly crop patches from the given image and then feed them as queries to the decoder. The model is pre-trained to detect these query patches from the original image. During the pre-training, we address two critical issues: multi-task learning and multi-query localization. (1) To trade off classification and localization preferences in the pretext task, we freeze the CNN backbone and propose a patch feature reconstruction branch which is jointly optimized with patch detection. (2) To perform multi-query localization, we introduce UP-DETR from single-query patch and extend it to multi-query patches with object query shuffle and attention mask. In our experiments, UP-DETR significantly boosts the performance of DETR with faster convergence and higher average precision on object detection, one-shot detection and panoptic segmentation. Code and pre-training models: https://github.com/dddzg/up-detr.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

EDTER: Edge Detection with Transformer (2022, methodology): “Recently, it is applied in conjunction with CNN in DETR [7] and the other variants [10, 27, 31, 58, 73].”
Meta-attention for ViT-backed Continual Learning (2022, background): “The transformer [46], a prevailing network architecture in nature language processing (NLP) [1, 9, 35], has received growing interest and achieved great accomplishment in the computer vision field, enjoying state-of-the-art performance on many visual tasks, including image classification [10, 28, 45”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, background): “Recent works have explored to apply Transformers to various vision tasks: image classification [5, 12], object detection [3, 59, 56, 8, 34], segmentation [4, 40], image enhancement [4, 45], image generation [27], video processing [58, 51], and 3D point cloud processing [54].”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “…a viable alternative to the convolutional neural networks (CNNs) in visual recognition tasks, such as classiﬁcation [11, 30], object detection [3, 45, 43, 8, 28], segmentation [33, 36], image enhancement [4, 40], image generation [24], video processing [42, 44] and 3D point cloud processing [12].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “In addition to image classification, Transformers have also been applied to other compute vision tasks, including object detection [3, 54, 50, 10], segmentation [41, 44], image enhancement [4, 46], image generation [28, 6], video processing [48, 53], and vision-language tasks [26, 35, 7, 33, 21, 20,”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “Since its promise firstly demonstrated in Vision Transformer (ViT) [23], we have witnessed a flourish of full-Transformer models for image classification [57, 63, 67, 44, 80, 59], object detection [9, 91, 84, 20] and semantic segmentation [61, 65].”
Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, background): “Self-supervised representation learning [2,14,27] constructs feature embedding spaces by designing pre-text tasks, such as solving jigsaw puzzles [34].”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, methodology): “classification [17, 37, 50, 51, 57, 58, 65, 67], object detection [5, 12, 74], and semantic segmentation [54, 63] to temporal tasks such as video classification [1, 3, 18] and object tracking [7, 41, 56].”
A Survey on Vision Transformer (2020, methodology): “UP-DETR [28] Unsupervised pre-training, random query patch detection arXiv 2020”
A Cookbook of Self-Supervised Learning (2023, background): “Recently, UP-DETR [Dai et al., 2021] and DETReg [Bar et al.”
Fast Convergence of DETR with Spatially Modulated Co-Attention (2021, background): “UP-DETR [5] propose a novel self-supervised loss to enhance the convergence speed.”
Efficient Self-supervised Vision Transformers for Representation Learning (2021, methodology): “Transformers have also been applied to other vision tasks, ranging from low-level tasks such as image generation [44, 8] and enhancement [7, 66], to high-level tasks such as object detection [3, 75, 73, 14] and segmentation [58, 61], and to vision-language tasks [41, 51, 13, 50, 35, 33, 74, 38].”
TransPose: Keypoint Localization via Transformer (2020, methodology): “The pre-training and fine-tuning for Transformer-based models have shown favorable results in NLP [17, 44] and recent vision models [18, 11, 16].”
Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency (2022, background): “Although there are studies on self-supervised representation learning for time series [7, 8, 31, 32] and self-supervised pre-training for images [33, 34, 35, 24], the intersection of these two areas, i.”
VidTr: Video Transformer Without Convolutions (2021, background): “To aggregate convolutional features for down-streaming tasks, e.g. object detection [5, 11], pose estimation [61], semantic segmentation [15] and action recognition [21].”
Uncertainty-Guided Transformer Reasoning for Camouflaged Object Detection (2021, background): “Recent works have tried to apply transformers to various vision tasks, such as object detection [2, 5, 66], image recognition [8,47], multi-object tracking [39,48], semantic segmentation [64], and human pose and mesh reconstruction [30].”