End-to-End Object Detection with Transformers
By Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko. European Conference on Computer Vision, 2020
19,102 citations recorded by the source corpus.
We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at this https URL.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
View Paper: https://arxiv.org/abs/2005.12872
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Quoc V. Le
- Dacheng Tao
- Yoshua Bengio
- Trevor Darrell
- Percy Liang
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Model Architectures
- Quoc V. Le
- Alan L. Yuille
- Jianfeng Gao
- Jiashi Feng
- Kaiming He
- Shuicheng Yan
- Chunhua Shen
- Dacheng Tao
Citations in the corpus
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…classification (Bello et al., 2019) or by further processing the output of a CNN using self-attention, e.g. for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classification (Wu et al., 2020), unsupervised object discovery…”
- A Review of Sparse Expert Models in Deep Learning (2022, background): “Some examples are image classification (Dosovitskiy et al., 2020), object detection (Carion et al., 2020), recommendation systems (Chen et al., 2019), speech recognition (Dong et al., 2018; Nakatani, 2019; Gulati et al., 2020).”
- VOLO: Vision Outlooker for Visual Recognition (2021, background): “including image classification [2], [3], [4], object detection [5], [6] and image processing like denoising, super-resolution and deraining [7].”
- An Empirical Study of Training Self-Supervised Vision Transformers (2021, background): “There have been continuous efforts on generalizing Transformers to computer vision [43, 5, 38, 48, 6, 15].”
- Training data-efficient image transformers & distillation through attention (2020, background): “More recently several researchers have proposed hybrid architecture transplanting transformer ingredients to convnets to solve vision tasks [6, 40].”
- Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “Several other works use attention to process or refine latent spaces that interface with the input/output data using task- or domain-specific architectures [12, 45, 92].”
- Perceiver: General Perception with Iterative Attention (2021, background): “…domain-agnostic analogue of models that stacks self-attention on top of convolutional feature maps to perform cheap but global processing on top or in conjunction with otherwise spatially localized convolutional feature maps (e.g. Carion et al. 2020; Locatello et al. 2020; Wang et al. 2021).”
- Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “Example applications include speaker diarization (Fujita et al., 2019), vector graphics generation (Carlier et al., 2020), object detection (Carion et al., 2020) and point cloud generation (Achlioptas et al., 2018).”
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, methodology): “[13] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.”
- FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The self-supervised pretraining paradigm has significantly advanced the state of the art across various domains, from natural language processing [6, 17–19, 23, 24, 28, 30, 61, 68, 73, 83–85], to computer vision [2, 5, 8, 12, 31, 33, 37, 59, 75, 104, 117], to speech recognition [4, 22, 42, 67, 119] ”
- Achieving Human Parity on Visual Question Answering (2021, background): “, 2021] builds upon the detection transformer (DETR) [Carion et al., 2020], and proposes an end-to-end grid-based VLP model for both V+L understanding and generation.”
- Pix2seq: A Language Modeling Framework for Object Detection (2021, methodology): “Per (Carion et al., 2020), we also experiment with the DC5 variant of ResNet (Li et al., 2017), which increases the resolution of its output feature map by a factor of two.”
- Scaling Vision Transformers (2021, background): “Attention-based Transformer architectures [44] have taken computer vision domain by storm [7, 15] and are becoming an increasingly popular choice in research and practice.”
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “A number of works use self-attention for images, either in combination with convolu-tions [7, 24, 63, 68] or even replacing them entirely.”
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “For these four frameworks, we utilize the same settings: multi-scale training [8, 52] (resizing the input such that the shorter side is between 480 and 800 while the longer side is at most 1333), AdamW [40] optimizer (initial learning rate of 0.”