RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder

By Cheng Chi, Fangyun Wei, Han Hu. 2020

73 citations recorded by the source corpus.

Existing object detection frameworks are usually built on a single format of object/part representation, i.e., anchor/proposal rectangle boxes in RetinaNet and Faster R-CNN, center points in FCOS and RepPoints, and corner points in CornerNet. While these different representations usually drive the frameworks to perform well in different aspects, e.g., better classification or finer localization, it is in general difficult to combine these representations in a single framework to make good use of each strength, due to the heterogeneous or non-grid feature extraction by different representations. This paper presents an attention-based decoder module similar as that in Transformer~\cite{vaswani2017attention} to bridge other representations into a typical object detector built on a single representation format, in an end-to-end fashion. The other representations act as a set of \emph{key} instances to strengthen the main \emph{query} representation features in the vanilla detectors. Novel techniques are proposed towards efficient computation of the decoder module, including a \emph{key sampling} approach and a \emph{shared location embedding} approach. The proposed module is named \emph{bridging visual representations} (BVR). It can perform in-place and we demonstrate its broad effectiveness in bridging other representations into prevalent object detection frameworks, including RetinaNet, Faster R-CNN, FCOS and ATSS, where about $1.5\sim3.0$ AP improvements are achieved. In particular, we improve a state-of-the-art framework with a strong backbone by about $2.0$ AP, reaching $52.7$ AP on COCO test-dev. The resulting network is named RelationNet++. The code will be available at this https URL.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Group-Free 3D Object Detection via Transformers (2021, methodology): “This is contrary to 2D image modeling, where such modeling tools mainly act as a chal-lenger or a complementary component to the mature grid modeling tools such as ConvNets variants [16, 32, 46] and RoI Align [2, 5].”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including image classification [31, 63, 60], object detection [11, 98, 22], semantic/instance segmentation [86], video understanding [8, 2, 33, 57], point cloud modeling [95, 41], 3D Object Rec”
A Survey on Vision Transformer (2020, background): “[23] propose the bridging visual representations (BVR) to combine different heterogeneous representations into a single one via a multihead attention module.”
Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model (2022, background): “Despite the remarkable success of DNNs [2, 9, 12, 24] in the computer vision field, they often require a large amount of annotated data in order to get satisfying results of object detection [3, 14, 20, 32].”
Rethinking Transformer-based Set Prediction for Object Detection (2020, methodology): “[4] uses a Transformer-like attention-based module to bridge different forms of representations.”