Sparse R-CNN: End-to-End Object Detection with Learnable Proposals

By Pei Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang and 1 more. 2020

1,499 citations recorded by the source corpus.

We present Sparse R-CNN, a purely sparse method for object detection in images. Existing works on object detection heavily rely on dense object candidates, such as k anchor boxes pre-defined on all grids of image feature map of size H × W. In our method, however, a fixed sparse set of learned object proposals, total length of N, are provided to object recognition head to perform classification and location. By eliminating HWk (up to hundreds of thousands) hand-designed object candidates to N (e.g. 100) learnable proposals, Sparse R-CNN completely avoids all efforts related to object candidates design and many-to-one label assignment. More importantly, final predictions are directly output without non-maximum suppression post-procedure. Sparse R-CNN demonstrates accuracy, run-time and training convergence performance on par with the well-established detector baselines on the challenging COCO dataset, e.g., achieving 45.0 AP in standard 3× training schedule and running at 22 fps using ResNet-50 FPN model. We hope our work could inspire re-thinking the convention of dense prior in object detectors. The code is available at: https://github.com/PeizeSun/SparseR-CNN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “For these four frameworks, we utilize the same settings: multi-scale training [8, 52] (resizing the input such that the shorter side is between 480 and 800 while the longer side is at most 1333), AdamW [40] optimizer (initial learning rate of 0.”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “ViLD [13] Mask R-CNN [15] X sentence X partial label partial label RegionCLIP [62] Faster RCNN [39] X sentence X partial label partial label FindIt [21] Faster RCNN [39] A sentence X partial label fine-tune MDETR [18] DETR [2] A,C word fine-tune zero-shot fine-tune DQ-DETR [46] DETR [2] A,C word X z”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, methodology): “In training, a multi-scale augmentation [9, 61] with the shorter side between 480 and 800 and the longer side of 1333 is used.”
MiniVLM: A Smaller and Faster Vision-Language Model (2020, neutral): “[56] or [12, 48, 3] to remove NMS.”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “In Table 3 To further verify the effectiveness of our proposed Focal Transformers, we follow [44] to train four different object detectors including Cascade R-CNN [7], ATSS [81], RepPoints [72] and Sparse R-CNN [55].”
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR (2022, methodology): “Besides DETR, Sun et al. (2021) proposed a sparse detector by learning boxes directly, which shares a similar anchor formulation with us, but it discards the Transformer structure and leverages hard ROI align for feature extraction.”
Instances as Queries (2021, methodology): “The recently proposed Sparse R-CNN [42] builds a query based setprediction framework upon R-CNN [19, 18, 39] based detector.”
Language as Queries for Referring Video Object Segmentation (2022, methodology): “DETR [4] introduces the new query-based paradigm [38, 55] for object detection, which employs a set of object queries as candidates and inputs them to the Transformer decoder.”
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer (2021, methodology): “All the models are trained under the same setting as in [28]: multi-scale training [4, 33] (resizing the input such that the shorter side is between 480 and 800 while the longer side is at most 1 , 333 ), AdamW [29] optimizer (initial learning rate of 0 .”
DetCo: Unsupervised Contrastive Learning for Object Detection (2021, methodology): “Sparse R-CNN is a recent strong end-to-end detector with high performance, and DetCo can further largely boost up Sparse R-CNN’s performance and achieved the new state of the arts on COCO with 46.5 AP.”
Efficient DETR: Improving End-to-End Object Detector with Dense Prior (2021, neutral): “Similar to the object queries in DETR, proposal features are fed to 6 dynamic instances interactive head [31] to interact with the region feature from feature pyramid networks (FPN) [19].”
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation (2021, methodology): “We consider the popular Cascade Mask-RCNN detection framework [4,19], and leverage multi-scale training [6, 35] (resizing the input to 896 × 896 ), AdamW optimizer [29] (with an initial learning rate as 3 × 10 − 3 ), weight decay as 1 × 10 − 4 , and a batch size of 256.”
DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries (2021, methodology): “SparseR-CNN [19] incorporates set prediction into a RCNN-style pipeline; it outperforms multi-stage object detection without NMS. OneNet [20] studies an interesting phenomenon: dense-based object detectors can be made NMS-free after they are equipped with a minimum-cost set loss.”
Anchor DETR: Query Design for Transformer-Based Detector (2022, neutral): “2015), Cascased RCNN (Cai and Vasconcelos 2018), Sparse RCNN (Sun et al. 2020), DETR (Carion et al.”
DETRs Beat YOLOs on Real-time Object Detection (2023, background): “Recently, the end-to-end Transformer-based detectors (DETRs) [4, 16, 22, 25, 33, 36, 41, 42] have received extensive attention from the academia due to their streamlined architecture and elimination of hand-crafted components.”