Densely Connected Convolutional Networks
By Gao Huang, Zhuang Liu, Kilian Q. Weinberger. 2016
44,055 citations recorded by the source corpus.
Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections—one between each layer and its subsequent layer—our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less memory and computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Dacheng Tao
- Percy Liang
- Sergey Levine
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Alan L. Yuille
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Alan L. Yuille
- Jianfeng Gao
- Jiashi Feng
- Ruslan Salakhutdinov
- Dacheng Tao
- Vladlen Koltun
Citations in the corpus
- Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “While sweeping across large hyperparameter spaces is common in Computer Vision (Huang et al., 2016b), such exploration would be prohibitively expensive for NMT models, limiting researchers to well-established architectures and hyperparameter choices.”
- U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “Following the success of DenseNet (Huang, Liu, and Weinberger 2016), we concatenate the input and output of each layer as the input of the next layer.”
- Universal Language Model Fine-tuning for Text Classification (2018, methodology): “…CV models (including object detection, classification, and segmentation) are rarely trained from scratch, but instead are fine-tuned from models that have been pretrained on ImageNet, MS-COCO, and other datasets (Sharif Razavian et al., 2014; Long et al., 2015a; He et al., 2016; Huang et al., 2017).”
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, background): “The precise effect of Batch Normalization on gradient propagation remains an area of further study.”
- A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “In addition to the previously examined setting of WRN with SGD, we add four more settings: two well known architectures (VGG and DenseNet), each trained with both SGD and Adam optimizers.”
- Rethinking ImageNet Pre-Training (2018, background): “They designed a specialized detector driven by deeply supervised networks [24] and dense connections [18].”
- Do Better ImageNet Models Transfer Better? (2018, methodology): “These networks encompassed widely used Inception architectures [67, 34, 68, 66]; ResNets [28, 26, 25]; DenseNets [31]; MobileNets [30, 59]; and NASNets [78].”
- Exploring the Limits of Weakly Supervised Pretraining (2018, neutral): “We believe our results will generalize to other architectures [24,25,26].”
- Fine-tuned Language Models for Text Classification (2018, methodology): “…CV models (including object detection, classification, and segmentation) are rarely trained from scratch, but instead are fine-tuned from models that have been pretrained on ImageNet, MS-COCO, and other datasets (Sharif Razavian et al., 2014; Long et al., 2015a; He et al., 2016; Huang et al., 2017).”
- AutoGrow: Automatic Layer Growing in Deep Convolutional Networks (2019, neutral): “For example, image classification accuracy keeps improving as the depth of network models grows [12, 15, 17, 32, 35].”
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019, methodology): “Depth (d): Scaling network depth is the most common way used by many ConvNets (He et al., 2016; Huang et al., 2017; Szegedy et al., 2015; 2016).”
- Exploring Self-Attention for Image Recognition (2020, background): “A succession of increasingly powerful convolutional architectures for image recognition followed [31, 30, 12, 16, 15].”
- Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, neutral): “96 DenseNet-201 (k=32) [41] 17.”
- Weight Standardization (2019, background): “BN stabilizes the training by controlling the first two moments of the distributions of the layer inputs in each mini-batch during training and is especially helpful for training very deep networks that have hundreds of layers [13, 14].”
- Group Normalization (2018, background): “As a result, many recent models [59, 20, 57, 24, 63] are trained with non-trivial batch sizes that are memory-consuming.”
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, background): “For the last 100 epochs the learning rate is linearly decayed to zero.”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, neutral): “[43] Huang, G.”
- Natural Adversarial Examples (2019, methodology): “xamples are selected to break a fixed model, in this case ResNet-50, but to black-box models they transfer reliably. For example, these images undermine models such as ResNeXt-50 [45] and DenseNet-121 [26], both of which have an IMAGENET-A accuracy rate of less than 3%. We examine ways to improve rob”
- Making Convolutional Networks Shift-Invariant Again (2019, methodology): “We present additional analysis in the supplementary material – using DenseNet (Huang et al., 2017 Baseline Anti-aliased (Rect-2) Anti-aliased (Tri-3) Anti-aliased (Bin-5) Figure 6.”