Swapout: Learning an ensemble of deep architectures

By Saurabh Singh, Derek Hoiem, D.A. Forsyth. 2016

156 citations recorded by the source corpus.

We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Recent advances in convolutional neural networks (2015, background): “[150] generalize dropout and networks with stochastic depth, which can be viewed as an ensemble of ResNets, Dropout ResNets, and Stochastic Depth ResNets.”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “Stochastic regularization techniques such as Dropout [35], Swapout [32], DropPath [20] or stochastic depth [14] superficially resemble our method, in that they obtain variable configurations by randomly dropping connectivity paths in a fixed network architecture.”
Convolutional Neural Fabrics (2016, methodology): “Finally, we analyze the number of response maps, parameters, and activations of fabrics in Section 3.3.”
PathNet: Evolution Channels Gradient Descent in Super Neural Networks (2017, methodology): “The following hyperparameters were investigated for PathNet: evaluation time T [1,10,50], mutation rate [0.1, 0.01, 0.001] and tournament size B [2, 10, 20].”
Understanding intermediate layers using linear classifier probes (2016, background): “papers seeking to understand better how they work (Veit et al., 2016; Larsson et al., 2016; Singh et al., 2016).”
Weight-averaged consistency targets improve semi-supervised deep learning results (2017, background): “Swapout (Singh et al., 2016) generalizes dropout and stochastic depth into a common method.”
Snapshot Ensembles: Train 1, get M for free (2017, methodology): “Finally, Swapout (Singh et al., 2016) is a stochastic training method that generalizes Dropout and Stochastic Depth.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “More recent results indicate that deep ensembles of small models can be more efficient than single large models on image classification [33, 44, 56, 35, 21], and we extend these findings to video classification.”
Wild Variational Approximations (2016, methodology): “dropout [25] and related variants [26, 27], have been shown as a variational inference method for network weights z = {W } [28, 29].”
Interleaved Structured Sparse Convolutional Neural Networks (2018, unknown): “D #Params(M ) C10 C100 Tiny ImageNet Swapout [33] 20 1.”
Deep Pyramidal Residual Networks (2016, unknown): “73 SwapOut v2 (width×4) [26] 7.”
PolyNet: A Pursuit of Structural Diversity in Very Deep Networks (2016, background): “Swapout [21] is a stochastic training method that generalizes dropout and stochastic depth by using random variables to control whether units are kept or dropped.”
Under review as a conference paper at ICLR 2020 many domain adaptation methods (2019, methodology): “The ensemble of multiple learners has widely been used to avoid the overfitting problem Singh et al. (2016); Han et al. (2017; 2016); Pi et al. (2016).”
Temporal Ensembling for Semi-Supervised Learning (2016, background): “…(Srivastava et al., 2014), dropconnect (Wan et al., 2013), or stochastic depth (Huang et al., 2016) regularization methods, and in swapout networks (Singh et al., 2016), where training always focuses on a particular subset of the network, and thus the complete network can be seen as an implicit…”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, background): “Contrarily, we find that zoneout outperforms stochastic depth and recurrent dropout in our experiments with RNNs.”
A review of modularization techniques in artificial neural networks (2019, background): “Swapout (Singh et al. 2016) generalizes dropout and stochastic depth, such that it is effectively sampling from a larger topological space, where a combination of dropping single units and whole layers is possible.”