Beyond Skip Connections: Top-Down Modulation for Object Detection

By Abhinav Shrivastava, Rahul Sukthankar, Jitendra Malik, Abhinav Kumar Gupta. 2016

327 citations recorded by the source corpus.

In recent years, we have seen tremendous progress in the field of object detection. Most of the recent improvements have been achieved by targeting deeper feedforward networks. However, many hard object categories such as bottle, remote, etc. require representation of fine details and not just coarse, semantic representations. But most of these fine details are lost in the early convolutional layers. What we need is a way to incorporate finer details from lower layers into the detection architecture. Skip connections have been proposed to combine high-level and low-level features, but we argue that selecting the right features from low-level requires top-down contextual information. Inspired by the human visual pathway, in this paper we propose top-down modulations as a way to incorporate fine details into the detection framework. Our approach supplements the standard bottom-up, feedforward ConvNet with a top-down modulation (TDM) network, connected using lateral connections. These connections are responsible for the modulation of lower layer filters, and the top-down network handles the selection and integration of contextual information and low-level features. The proposed TDM architecture provides a significant boost on the COCO testdev benchmark, achieving 28.6 AP for VGG16, 35.2 AP for ResNet101, and 37.3 for InceptionResNetv2 network, without any bells and whistles (e.g., multi-scale, iterative box refinement, etc.).

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Mask R-CNN (2017, methodology): “0 points box AP over the best previous single model entry from [36] (which used Inception-ResNet-v2-TDM).”
Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection (2019, neutral): “8 TDM [53] trainval Inception-ResNet-v2-TDM 36.”
Soft-NMS — Improving Object Detection with One Line of Code (2017, neutral): “ requires re-training of models which leads to a little improvement in performance, it will also be ignored and is typically referred to as “bells-and-whistles” in the recent work on object detection [15,16,25]. However, if a simple module can improve performance without requiring any re-training of”
Focal Loss for Dense Object Detection (2017, methodology): “0 Faster R-CNNw TDM [30] Inception-ResNet-v2-TDM 36.”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “…600 × 600 29.6% - ResNet-101-FPN [24] 800 (short side) - 36.2% Inception-ResNet-v2 (G-RMI) [18] 600 × 600 35.7% 35.6% Inception-ResNet-v2 (TDM) [31] 600 × 1000 37.3% 36.8% NASNet-A (6 @ 4032) 800 × 800 41.3% 40.7% NASNet-A (6 @ 4032) 1200 × 1200 43.2% 43.1% ResNet-101-FPN (RetinaNet) [25] 800…”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, methodology): “Encoder-decoder: The encoder-decoder networks have been successfully applied to many computer vision tasks, including human pose estimation [53], object detection [45, 66, 19], and semantic segmentation [49, 54, 61, 3, 43, 59, 57, 33, 76, 20].”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, background): “This type of model is also explored in the context of object detection [47, 66].”
Path Aggregation Network for Instance Segmentation (2018, background): “Both TDM [56] and FPN [35] augmented a top-down path with lateral connections for object detection.”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, background): “In the future, we have plans to relax this search space even further to include U-net architectures [63, 45, 70], where layer l may receive input from one more layer preceding l in addition to l − 1.”
YOLOv3: An Incremental Improvement (2018, neutral): “0 Faster R-CNN w TDM [20] Inception-ResNet-v2-TDM 36.”
CornerNet: Detecting Objects as Paired Keypoints (2018, background): “In fact, many current detectors (Shrivastava et al., 2016; Fu et al., 2017; Lin et al., 2016; Kong et al., 2017) already adopted networks similar to the hourglass network.”
Single-Shot Refinement Neural Network for Object Detection (2017, methodology): “2 Faster R-CNN w TDM [42] trainval Inception-ResNet-v2-TDM 36.”
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features (2017, methodology): “Our ResNet-101 based model even without scale augmentation during training performs better than G-RMI [35] and TDM [66] which employ more expensive yet powerful Inception-ResNet-v2 [68] as feature extractor.”
CenterNet: Keypoint Triplets for Object Detection (2019, neutral): “2 - - - - Faster R-CNN w/ TDM [38] Inception-ResNet-v2 36.”
Scale-Aware Trident Networks for Object Detection (2019, background): “TDM [37] and FPN [25] further introduce a top-down pathway and lateral connections to enhance the semantic representation of low-level features at bottom layers.”
A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection (2017, methodology): “The second area of research has been to use contextual reasoning, proxy tasks for reasoning and other top-down mechanisms for improving representations for object detection [1, 5, 17, 24, 32, 34, 45].”
An Analysis of Scale Invariance in Object Detection - SNIP (2017, background): “For example, features from the layers near to the input, referred to as shallow(er) layers, are combined with deeper layers for detecting small object instances [23, 34, 1, 13, 27], dilated/deformable convolution is used to increase receptive fields for detecting large objects [32, 7, 37, 8], indepe”
Deep Feature Pyramid Reconfiguration for Object Detection (2018, methodology): “To overcome the disadvantage of SSD and make the networks more robust to object scales, recent works (e.g., FPN [30], DSSD [15], RON [26] and TDM [44]) propose to combine low-resolution and semantically-strong features with high-resolution and semantically-weak features via lateral connections in a…”