Convolutional Networks with Dense Connectivity

By Gao Huang, Zhuang Liu, Geoff Pleiss, L. V. D. van der Maaten, Kilian Q. Weinberger. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

549 citations recorded by the source corpus.

Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with <inline-formula><tex-math notation="LaTeX">$L$</tex-math><alternatives><mml:math><mml:mi>L</mml:mi></mml:math><inline-graphic xlink:href="huang-ieq1-2918284.gif"/></alternatives></inline-formula> layers have <inline-formula><tex-math notation="LaTeX">$L$</tex-math><alternatives><mml:math><mml:mi>L</mml:mi></mml:math><inline-graphic xlink:href="huang-ieq2-2918284.gif"/></alternatives></inline-formula> connections—one between each layer and its subsequent layer—our network has <inline-formula><tex-math notation="LaTeX">$\frac{L(L+1)}{2}$</tex-math><alternatives><mml:math><mml:mfrac><mml:mrow><mml:mi>L</mml:mi><mml:mo>(</mml:mo><mml:mi>L</mml:mi><mml:mo>+</mml:mo><mml:mn>1</mml:mn><mml:mo>)</mml:mo></mml:mrow><mml:mn>2</mml:mn></mml:mfrac></mml:math><inline-graphic xlink:href="huang-ieq3-2918284.gif"/></alternatives></inline-formula> direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, encourage feature reuse and substantially improve parameter efficiency. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less parameters and computation to achieve high performance.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/6f42a3843b72155bd375eac2c66516bd19b530ec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

On the Integration of Self-Attention and Convolution (2021, background): “recognition [19, 23], semantic segmentation [9] and object detection [38], and achieve state-of-the-art performances on various benchmarks.”
Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (2020, methodology): “With ResNets and DenseNets, GFNet reduces the number of required Multiply-Adds for the given test accuracy by approximately 2− 3× times.”
On Feature Normalization and Data Augmentation (2020, background): “Fueled by these successes, signiﬁcant effort has gone into the search for ever more powerful and bigger neural network architectures (Krizhevsky et al., 2012; He et al., 2015; Zoph & Le, 2016; Huang et al., 2019; Vaswani et al., 2017).”
Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection (2022, methodology): “…of the k th pixel and H x is the histogram of pixel value i , we deﬁne the pixel loss L pixel as: where 3 https://github.com/ultralytics/YOLOv5 We employ a 5-layer dense block [6] as G to extract common features, and then use a merge block with three convolutional layers for feature aggregation.”
Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length (2021, methodology): “For all the three datasets, we adopt the same data pre-processing and data augmentation policy as [18, 23, 22].”
Vision Transformer with Deformable Attention (2022, background): “Compared to their CNN counterparts [18,19], Transformer-based models have larger receptive ﬁelds and excel at modeling long-range dependencies, which are proved to achieve superior performance in the regime of a large amount of training data and model parameters.”
Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks (2021, methodology): “For the CIFAR10 domain, we test on a pre-trained DenseNet121 model [Huang et al., 2019, Phan, 2021] to highlight the fact that SCOD can be applied to augment any pre-trained model, independent of training methodology, with uncertainty estimates.”
P2T: Pyramid Pooling Transformer for Scene Understanding (2021, methodology): “DenseNets [5] introduce dense connections that connect each layer to all its subsequent layers for easing optimization.”
Systematic Evaluation of Privacy Risks of Machine Learning Models (2020, methodology): “The CIFAR100 classifier is a publicly available pre-trained model,4 with the DenseNet architecture [16, 17].”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, methodology): “For all the three datasets, we adopt the same data pre-processing and data augmentation policy as [19, 24, 23].”