Deformable ConvNets V2: More Deformable, Better Results

By Xizhou Zhu, Han Hu, Stephen Lin, Jifeng Dai. 2018

2,639 citations recorded by the source corpus.

The superior performance of Deformable Convolutional Networks arises from its ability to adapt to the geometric variations of objects. Through an examination of its adaptive behavior, we observe that while the spatial support for its neural features conforms more closely than regular ConvNets to object structure, this support may nevertheless extend well beyond the region of interest, causing features to be influenced by irrelevant image content. To address this problem, we present a reformulation of Deformable ConvNets that improves its ability to focus on pertinent image regions, through increased modeling power and stronger training. The modeling power is enhanced through a more comprehensive integration of deformable convolution within the network, and by introducing a modulation mechanism that expands the scope of deformation modeling. To effectively harness this enriched modeling capability, we guide network training via a proposed feature mimicking scheme that helps the network to learn features that reflect the object focus and classification power of R-CNN features. With the proposed contributions, this new version of Deformable ConvNets yields significant performance gains over the original model and produces leading results on the COCO benchmark for object detection and instance segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Another related method is deformable convolution [24,100,28], where each point attends to a few points dynamically on an image.”
Local Relation Networks for Image Recognition (2019, methodology): “This has been done to enlarge the receptive field, such as through atrous/dilated convolution [4, 33], and to enhance the ability to model geometric deformation, via active [14] and deformable convolution [6, 36].”
Objects as Points (2019, methodology): “Sophisticated two-stage detectors [31,35,48,63] are more accurate, but also slower.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, background): “In addition to these architectural advances, there has also been much work on improving individual convolution layers, such as depthwise convolution [64] and deformable convolution [17, 75].”
K-Net: Towards Unified Image Segmentation (2021, background): “Previous works [16, 18, 25, 26, 71] explore different kinds of dynamic kernels to improve the flexibility and performance of models.”
Dynamic Neural Networks: A Survey (2021, methodology): “TABLE 2 Kernel shape adaptation by dynamically sampling feature pixels [110], [111] or convolutional weights [112].”
Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, methodology): “On the other hand, as discussed in Zhu et al. (2019a), there are variants of convolution, such as deformable convolution (Dai et al., 2017; Zhu et al., 2019b) and dynamic convolution (Wu et al., 2019), that also can be viewed as self-attention mechanisms.”
In Defense of Grid Features for Visual Question Answering (2020, methodology): “To reduce the effect of low resolutions when training the detector with features pooled from C5 (C5 has stride 32, whereas C4 has 16), the stride-2 layers are replaced with stride-1 layers, and the remaining layers are dilated with a factor of 2 [55].”
Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, neutral): “• conv: 3× 3 convolution; • sep: 3× 3 separable convolution; • dil: 3× 3 convolution with dilation rate of 2; • def: 3× 3 deformable convolution v2 [45]; Figure 2: Outer layer width search.”
DetNAS: Backbone Search for Object Detection (2019, background): “The importance of backbones is also demonstrated in DetNet [11], Deformable ConvNets v2 [29], ThunderNet [21] in real-time object detection, and HRNet [24] in keypoint detection.”
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, methodology): “We additionally provide results for Faster R-CNN and Mask R-CNN models with deformable convolutions [Dai et al., 2017, Zhu et al., 2018] in Appendix D. Models were evaluated using the mmdetection toolbox [Chen et al., 2019b]; all models were trained and tested with standard hyperparameters.”
Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection (2020, background): “Further, Deformable Convolutional Networks (DCN) [36] consistently boost the performances over ResNe(X)t backbones, where GFL with ResNeXt-101-32x4d-DCN obtains state-of-the-art 48.2% AP at 10 FPS. Fig.”
MMDetection: Open MMLab Detection Toolbox and Benchmark (2019, background): “• DCNv2 [42]: modulated deformable operators, proposed in 2018.”
ResNeSt: Split-Attention Networks (2020, neutral): “main the same. We re-train the baseline with the same setting described above, but with the standard ResNet. All our ResNeSt: Split-Attention Networks 13 ork Method Backbone box mAP% mask mAP% DCV-V2 [72] ResNet50 42.7 37.0 HTC [4] ResNet50 43.2 38.0 Mask-RCNN [22] ResNet101 [5] 39.9 36.1 Cascade-RC”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, methodology): “Moreover, prevalent modules like deformable convolution [9, 47]”
Dynamic Head: Unifying Object Detection Heads with Attentions (2021, background): “[37] reformulated the offset by introducing a learned feature amplitude and further improved its ability.”
Scale-Aware Trident Networks for Object Detection (2019, background): “of the major branch to incorporate objects of all scales, the performance of Trident Method Backbone AP AP 50 AP 75 AP s AP m AP l Cascade R-CNN [5] ResNet-101-FPN 42.8 62.1 46.3 23.7 45.5 55.2 DCNv2 [44] ResNet-101-DeformableV2 46.0 67.9 50.8 27.8 49.1 59.5 DCR [9] ResNet-101-FPN-Deformable 43.1 66”
Efficient Neural Architecture Transformation Searchin Channel-Level for Object Detection (2019, methodology): “Following the regime mentioned in DCNv2 [44], we apply architecture transformation search only on blocks of stage-3,4,5 in backbone.”