MegDet: A Large Mini-Batch Object Detector

By Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu. 2017

336 citations recorded by the source corpus.

The development of object detection in the era of deep learning, from R-CNN [11], Fast/Faster R-CNN [10, 31] to recent Mask R-CNN [14] and RetinaNet [24], mainly come from novel network, new framework, or loss design. However, mini-batch size, a key factor for the training of deep neural networks, has not been well studied for object detection. In this paper, we propose a Large Mini-Batch Object Detector (MegDet) to enable the training with a large minibatch size up to 256, so that we can effectively utilize at most 128 GPUs to significantly shorten the training time. Technically, we suggest a warmup learning rate policy and Cross-GPU Batch Normalization, which together allow us to successfully train a large mini-batch detector in much less time (e.g., from 33 hours to 4 hours), and achieve even better accuracy. The MegDet is the backbone of our submission (mmAP 52.5%) to COCO 2017 Challenge, where we won the 1st place of Detection task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Momentum Contrast for Unsupervised Visual Representation Learning (2019, methodology): “We train with multiple GPUs and perform BN on the samples independently for each GPU (as done in common practice).”
Rethinking ImageNet Pre-Training (2018, methodology): “Learning curves of AP on COCO val2017 using Mask R-CNN with R50-FPN and SyncBN [34, 27] (that synchronizes batch statistics across GPUs).”
Weight Standardization (2019, background): “To alleviates BN’s issue when the batches become small, BR [16] constrains the statistics of BN within a certain range to reduce their drift when the batch size is not suf-ﬁciently large, and [30] proposes synchronized BN which synchronizes statistics across multiple GPUs through engi-2 neering.”
Group Normalization (2018, methodology): “(2018), and BN is synced across GPUs (Peng et al. 2018) and is not frozen.”
VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “We choose models trained with a similar compute budget as ours (8 GPUs, 200 ImageNet epochs).”
Unified Perceptual Parsing for Scene Understanding (2018, neutral): “It is worth noting that batch size has proven to be important to generate accurate statistics for tasks like classiﬁcation [38], semantic segmentation [16] and object detection [39].”
Cascade R-CNN: High Quality Object Detection and Instance Segmentation (2019, background): “6 mAP), which won the COCO challenge in 2017 and uses many other enhancements [44].”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Based on common practice in semantic [4, 40] and instance segmentation [29, 24], we expect the performance to be further improved with BN layers fine-tuned.”
Panoptic Feature Pyramid Networks (2019, methodology): “Recent innovations include Cascade R-CNN [7], deformable convolution [15], and sync batch norm [45].”
Hybrid Task Cascade for Instance Segmentation (2019, methodology): “Synchronized Batch Normalization [34, 28] is used in the backbone and heads.”
What makes for good views for contrastive learning (2020, background): “Therefore, we fine-tune the backbone with Synchronized BN (SyncBN [54]) and add SyncBN to newly initialized layers (e.g., FPN [40]).”
EfficientDet: Scalable and Efficient Object Detection (2019, methodology): “We omit ensemble and test-time multi-scale results [23, 7].”
DetNAS: Backbone Search for Object Detection (2019, methodology): “To this end, we replace the conventional BN with Synchronized Batch Normalization (SyncBN) [19] during supernet training.”
DropBlock: A regularization method for convolutional networks (2018, background): “Recently, a few papers tried to address the issue using novel model architecture [33], large minibatch size [34], and better normalization layer [35].”
MMDetection: Open MMLab Detection Toolbox and Benchmark (2019, methodology): “• SyncBN [25]: synchronized batch normalization across GPUs, we adopt the ofﬁcial implementation by PyTorch.”
Deep High-Resolution Representation Learning for Visual Recognition (2019, background): “Table 11 reports the comparison of our network to state-of-the-art single-model object detectors on COCO test-dev without using multi-scale training and multi-scale testing that are done in [85], [97], [110], [115], [127], [128].”
Context Encoding for Semantic Segmentation (2018, methodology): “Concurrent work also implement SyncBN by first calculating the global mean and then the variance, which requires synchronizing twice in each iteration [34, 43].”