Relation Networks for Object Detection

By Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, Yichen Wei. 2017

1,349 citations recorded by the source corpus.

Although it is well believed for years that modeling relations between objects would help object recognition, there has not been evidence that the idea is working in the deep learning era. All state-of-the-art object detection systems still rely on recognizing object instances individually, without exploiting their relations during learning. This work proposes an object relation module. It processes a set of objects simultaneously through interaction between their appearance feature and geometry, thus allowing modeling of their relations. It is lightweight and in-place. It does not require additional supervision and is easy to embed in existing networks. It is shown effective on improving object recognition and duplicate removal steps in the modern object detection pipeline. It verifies the efficacy of modeling object relations in CNN based detection. It gives rise to the first fully end-to-end object detector.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…maps for image classiﬁcation (Bello et al., 2019) or by further processing the output of a CNN using self-attention, e.g. for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classiﬁcation (Wu et al., 2020), unsupervised…”
End-to-End Object Detection with Transformers (2020, methodology): “prove their performance. More recent detectors [37,23,53] use non-unique assignment rules between ground truth and predictions together with an NMS. Learnable NMS methods [16,4] and relation networks [17] explicitly model relations between dierent predictions with attention. Using direct set losses,”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “transformers, attention is also widely used in different forms in computer vision models [21, 20, 41, 44, 46, 40, 28, 18, 19, 1, 49, 30].”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Attention has since been extended to vision, giving significant boosts to image classification [6], object detection [37], semantic segmentation [42], video classification [87], and adversarial defense [89].”
Local Relation Networks for Image Recognition (2019, background): “In [12], relationships between object proposals are modeled, which leads to improved accuracy as well as the first fully end-to-end object detector.”
Training data-efficient image transformers & distillation through attention (2020, background): “Nevertheless hybrid architectures that combine convnets and transformers, including the self-attention mechanism, have recently exhibited competitive results in image classification [53], detection [6, 26], video processing [42, 50], unsupervised object discovery [33], and unified text-vision tasks ”
Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection (2019, neutral): “[18] trainval35k ResNet-101 39.”
Cascade R-CNN: High Quality Object Detection and Instance Segmentation (2019, background): “G-RMI [28], the recent Deformable R-FCN [8], RetinaNet [36], Mask R-CNN [24], RelationNet [27], DetNet [34], CornerNet [32], etc.”
Panoptic Segmentation (2018, background): “To produce algorithm outputs for PS, we start from the non-overlapping instance segments from the NMS-like procedure described previously.”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “A number of works use selfattention for images, either in combination with convolutions [7, 19, 56, 61] or even replacing them entirely.”
VL-BERT: Pre-training of Generic Visual-Linguistic Representations (2019, methodology): “Following the practice in Relation Networks (Hu et al., 2018), the 4-d vector is embedded into a high-dimensional representation (of 2048-d in paper) by computing sine and cosine functions of different wavelengths.”
Attention on Attention for Image Captioning (2019, background): “Several works extend the idea of employing self-attention to some tasks [40, 19] in computer vision, which inspires us to apply self-attention to image captioning to model relationships among objects in an image.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, background): “The selfattention layers can complement backbones [61, 7, 3, 65, 21, 68, 51] or head networks [29, 24] by providing the capability to encode distant dependencies or heterogeneous interactions.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, background): “Both classes of methods have boosted various vision tasks such as image classification [18, 5, 72, 37, 57, 89, 28], object detection [91, 80, 72, 36, 10, 108], semantic segmentation [15, 106, 39, 29, 109, 107], video recognition [91, 18], image generation [73, 35], and panoptic segmentation [89].”
Relational inductive biases, deep learning, and graph networks (2018, background): “…Gilmer et al., 2017), to predict traﬃc on roads (Li et al., 2017; Cui et al., 2018), to classify and segment images and videos (Wang et al., 2018c; Hu et al., 2017) and 3D meshes and point clouds (Wang et al., 2018d), to classify regions in images (Chen et al., 2018a), to perform…”
Relational Deep Reinforcement Learning (2018, background): “This approach builds oﬀ previous work suggesting that relational computations needn’t necessarily be biased by entities’ spatial proximity [20, 10, 21, 11, 13, 22], and may also proﬁt from iterative structured reasoning [23, 24, 25, 26].”
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “Recently, deep neural networks (DNNs) became the dominant paradigm in various real-world applications, for example, image classiﬁcation [12, 18, 17, 16, 19], semantic segmentation [43, 32, 27], object detection [39, 15, 38, 46, 35].”
Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection (2018, methodology): “Finally, we plan to apply our uncertainty estimation to active learning and object tracking to improve bounding box predictions, or to incorporate these uncertainty estimations as additional knowledge to the training phase (e.g. [33]).”
Deformable ConvNets V2: More Deformable, Better Results (2018, background): “Recently, the concurrent works of [23] and [36] successfully extend relation networks and attention modules to the image domain, for modeling long-range object-object and pixel-pixel relations, respectively.”
Dual Attention Network for Scene Segmentation (2018, background): “For example, the work (Hu et al. 2017) proposes an object relation module to model the relationships among a set of objects, which improves object recognition.”