FractalNet: Ultra-Deep Neural Networks without Residuals

By Gustav Larsson, Michael Maire, Gregory Shakhnarovich. 2016

1,061 citations recorded by the source corpus.

We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals. These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks. Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. Such regularization allows extraction of high-performance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (2015, background): “The precise effect of Batch Normalization on gradient propagation remains an area of further study.”
Benchmarking Detection Transfer Learning with Vision Transformers (2021, background): “However, we found that drop path regularization [24, 22] is highly effective for ViT back-bones and therefore we include it ( e.g ., it improves from-scratch training by up to 2 AP box ).”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “We also use drop path [38] with a probability of 0.”
Designing Network Design Spaces (2020, methodology): “We emphasize that R EG N ET models use our basic 100 epoch schedule with no regularization except weight decay, while most mobile networks use longer schedules with various enhancements, such as deep supervision [16], Cutout [4], DropPath [14], AutoAugment [2], and so on.”
On Network Design Spaces for Visual Recognition (2019, methodology): “As a sanity check, we perform point comparisons using larger models and the exact training settings from DARTS [21] which uses a 600 epoch schedule with deep supervision [18], Cutout [5], and modified DropPath [17].”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “In DropPath [33], each path in the cell is stochastically dropped with some fixed probability during training.”
Regularized Evolution for Image Classifier Architecture Search (2018, background): “Until recently, most state-of-the-art (SOTA) convolutional architectures have been manually designed by human experts (Krizhevsky et al., 2012; Ciregan et al., 2012; Lin et al., 2014; Sermanet et al., 2014; He et al., 2014; Chatfield et al., 2014; Simonyan & Zisserman, 2015; Karpathy et al., 2014; S”
Efficient Neural Architecture Search via Parameter Sharing (2018, background): “5 on the skip connections, and to drop-path on the operations (Zoph et al., 2018; Larsson et al., 2017).”
Shake-Shake regularization (2017, methodology): “This method can be seen as a form of drop-path (Larsson et al., 2016) where residual branches are scaled-down instead of being completely dropped (i.”
Hierarchical Representations for Efficient Architecture Search (2017, background): “…state-of-the-art convolutional neural networks are going beyond deep, chain-structured layout (Simonyan & Zisserman, 2014; He et al., 2016a) towards increasingly more complex, graph-structured topologies (Szegedy et al., 2015; 2016; 2017; Larsson et al., 2016; Xie et al., 2016; Huang et al., 2016).”
PolyNet: A Pursuit of Structural Diversity in Very Deep Networks (2016, methodology): “Two of such examples are DenseNet [12], which adds shortcuts densely to the building blocks, and FractalNet [17], which uses a truncated fractal layout.”
How to train your MAML (2018, methodology): “Annealing the learning rate using either step or cosine functions has proven crucial to achieving state of the art generalization performance in a multitude of settings (Loshchilov & Hutter, 2016; He et al., 2016; Larsson et al., 2016; Huang et al., 2017).”
Surrogate-Assisted Evolutionary Deep Learning Using an End-to-End Random Forest-Based Performance Predictor (2020, background): “The ﬁrst category covers DenseNet [50], ResNet [54], Maxout [57], VGG [58], Network in Network [59], Highway Network [60], All-CNN [61], FractaNet [62].”
On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “In the ﬁnal-retrain phase, the optimal network architecture will be trained with additional regularization techniques, e.g., Shake-Shake (Gastaldi, 2017), DropPath (Larsson et al., 2017), and Cutout (DeVries and Taylor, 2017).”