Aggregated Residual Transformations for Deep Neural Networks
By Saining Xie, Ross B. Girshick, Piotr Doll'ar, Zhuowen Tu, Kaiming He. 2016
11,783 citations recorded by the source corpus.
We present a simple, highly modularized network architecture for image classification. Our network is constructed by repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has only a few hyper-parameters to set. This strategy exposes a new dimension, which we call cardinality (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width. On the ImageNet-1K dataset, we empirically show that even under the restricted condition of maintaining complexity, increasing cardinality is able to improve classification accuracy. Moreover, increasing cardinality is more effective than going deeper or wider when we increase the capacity. Our models, named ResNeXt, are the foundations of our entry to the ILSVRC 2016 classification task in which we secured 2nd place. We further investigate ResNeXt on an ImageNet-5K set and the COCO detection set, also showing better results than its ResNet counterpart. The code and models are publicly available online.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Jianfeng Gao
- Mohit Bansal
- Yoshua Bengio
- Dacheng Tao
- Trevor Darrell
- Percy Liang
- Sergey Levine
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Alan L. Yuille
In Model Architectures
- Quoc V. Le
- Alan L. Yuille
- Jianfeng Gao
- Jiashi Feng
- Yoshua Bengio
- Shuicheng Yan
- Christoph Feichtenhofer
- Dacheng Tao
Citations in the corpus
- Mask R-CNN (2017, methodology): “It benefits from deeper networks (50 vs. 101) and advanced designs including FPN and ResNeXt3.”
- Momentum Contrast for Unsupervised Visual Representation Learning (2019, neutral): “net [23], RX50 is ResNeXt-50-32×8d [62].”
- Rethinking ImageNet Pre-Training (2018, methodology): “We investigate Mask R-CNN [13] with ResNet [17] or ResNeXt [49] plus Feature Pyramid Network (FPN) [26] backbones.”
- Exploring the Limits of Weakly Supervised Pretraining (2018, methodology): “The figure shows results for models pretrained to predict 1.5k hashtags (dashed lines) or 17k hashtags (solid lines) for ResNeXt-101 models with three different capacities (represented by different colors).1 The four panels correspond to ImageNet target tasks with three different number of classes (”
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “#Params ResNet-152(Heetal.,2016) 77.8% 60M EfficientNet-B1 78.8% 7.8M ResNeXt-101(Xieetal.,2017) 80.9% 84M EfficientNet-B3 81.1% 12M SENet(Huetal.,2018) 82.7% 146M NASNet-A(Zophetal.,2018) 82.7% 89M EfficientNet-B4 82.6% 19M GPipe(Huangetal.,2018) † 84.3% 556M EfficientNet-B7 84.4% 66M † Notplotted Figu”
- Are we done with ImageNet? (2020, methodology): “We start by assembling a set of 19 models [3, 27, 4, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37] which vary along a number of dimensions, including architecture, training objective and usage of external data (see appendix A).”
- Exploring Self-Attention for Image Recognition (2020, background): “Momentum and weight decay are set to 0.9 and 1e-4, respectively [12, 38, 11].”
- Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, neutral): “31 ResNeXt-101 (32x4d) [90] 17.”
- Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “gree of noise (20% precision errors) in the labels. An even larger, noisily labelled dataset of 3.5B Instagram images is used in [36]. This increase in dataset size and an improved model architecture [62] lead to better results when transferring to ILSVRC-2012. We Big Transfer (BiT): General Visual ”
- Stand-Alone Self-Attention in Vision Models (2019, background): “The field of deep learning has in turn largely shifted toward the design of architectures of CNNs for improving the performance on image recognition [11–16], object detection [17–19] and image segmentation [20–22].”
- Local Relation Networks for Image Recognition (2019, background): “Our experimental settings and hyper-parameters mostly follow [31].”
- Attention Augmented Convolutional Networks (2019, methodology): “In the subsequent experiments, we test Attention Augmentation on standard computer vision architectures such as ResNets [14, 47, 13], and MnasNet [42] on the CIFAR100 [22], ImageNet [9] and COCO [25] datasets.”
- Weight Standardization (2019, methodology): “Especially for ResNeXt-101, the performance difference is large between GN and BN.”
- Group Normalization (2018, background): “The work of ResNeXt (Xie et al. 2017) investigates the trade-off between depth, width, and groups, and it suggests that a larger number of groups can improve accuracy under similar computational cost.”
- Non-local Neural Networks (2017, methodology): “We see that a single non-local block improves all R50/101 and X152 baselines, on all metrics involving detection and segmentation.”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “resnext101_32x4d [103].”
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, methodology): “Model size evaluations use ResNets or ResNeXts [36] of varying sizes.”
- VirTex: Learning Visual Representations from Textual Annotations (2020, background): “Visual Backbone Ablations: Bigger visual backbones tend to show improvements on many vision tasks [2, 9, 103].”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “We evaluate two image encoders: 1) standard ResNet-152 [30] convolutional features from res-5c with average pooling (Image-Grid) 2) features from fc6 layer of Faster-RCNN [60] with ResNeXt152 as its backbone [86].”