Dynamic Capacity Networks

By Amjad Almahairi, Nicolas Ballas, Tim Cooijmans, Yin Zheng, Hugo Larochelle, Aaron C. Courville. 2015

109 citations recorded by the source corpus.

We introduce the Dynamic Capacity Network (DCN), a neural network that can adaptively assign its capacity across different portions of the input data. This is achieved by combining modules of two types: low-capacity sub-networks and high-capacity sub-networks. The low-capacity sub-networks are applied across most of the input, but also provide a guide to select a few portions of the input on which to apply the high-capacity sub-networks. The selection is made using a novel gradient-based attention mechanism, that efficiently identifies input regions for which the DCN's output is most sensitive and to which we should devote more capacity. We focus our empirical evaluation on the Cluttered MNIST and SVHN image datasets. Our findings indicate that DCNs are able to drastically reduce the number of computations, compared to traditional convolutional neural networks, while maintaining similar or even better performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

In Model Architectures

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “…forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015).”
Dynamic Neural Networks: A Survey (2021, background): “The aforementioned dynamic additional refinement approaches [84], [149] are mainly developed for image classification.”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Finally, our approach is also related to conditional computation methods [34,35,36,37,38,39,40,41,42,43], which learn to drop out blocks of units.”
Spatially Adaptive Computation Time for Residual Networks (2016, methodology): “Additionally, choosing the glimpse positions requires designing a separate prediction network or a heuristic procedure [1].”
Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification (2016, methodology): “Methods for compressing and accelerating convolutional networks include knowledge distillation [12, 29], low-rank-factorization [14, 39, 33], pruning and quantization [10, 27], structured matrices [6, 35, 9], and dynamic capacity networks [2].”
Recurrent Neural Networks (2018, background): “Conditional computation has been shown to gradually increase model capacity without proportional increases in computational cost by exploiting certain computation paths for each input (Bengio et al., 2013; Liu & Deng, 2017; Almahairi et al., 2016; McGill & Perona, 2017; Shazeer et al., 2017).”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “1) Sparse: The sparse gating functions activate a selected subset of experts for processing each individual input token, which can be considered as a form of conditional computation [134], [135], [136].”
EagleEye: Attack-Agnostic Defense against Adversarial Inputs (Technical Report) (2018, background): “The magnification operation is loosely based on attention mechanisms [1, 16], inspired by that human’s vision automatically focuses on certain regions of an image with “high resolution” while perceiving surrounding regions in “low resolution”.”
Learning Instance-wise Sparsity for Accelerating Deep Models (2019, methodology): “[Almahairi et al. , 2016; Dong et al. , 2017; Ren et al. , 2018] applied the dynamic strategy on the activations of feature maps in neural networks.”
Adaptive Neural Trees (2018, background): “…(Bengio et al., 2013; Davis & Arel, 2013; Bengio et al., 2015; Shazeer et al., 2017) to adapting the required computation to the difﬁculty of the input and task (Bengio et al., 2015; Almahairi et al., 2016; Teerapittayanon et al., 2016; Graves, 2016; Figurnov et al., 2017; Veit & Belongie, 2017).”
Energy-efficient Amortized Inference with Cascaded Deep Classifiers (2017, methodology): “In addition to static techniques, Dynamic Capacity Network [Almahairi et al. , 2016] adaptively assigns its capacity across different portions of the input data by using a low and a high capacity sub-network.”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, background): “Conditional computation has been shown to allow gradual increases in model capacity without a proportional increases in computational cost by exploiting certain computation paths for each input [3, 16, 1, 17, 20].”
Runtime Neural Pruning (2017, methodology): “Dynamic Capacity Network [1] used a specially designed method to calculate a saliency map for control execution.”
Learned Contextual Feature Reweighting for Image Geo-Localization (2017, background): “Because task-relevant information is not generally uniformly distributed throughout an image [1], focusing on “interesting” areas, as opposed to “irrelevant” or even “distracting” areas, can often achieve better performance [51, 28, 13].”
Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference (2019, methodology): “A first set of methods [1, 32, 40] uses glimpses to only process regions of interest.”
Enriched Deep Recurrent Visual Attention Model for Multiple Object Recognition (2017, background): “[1] — Dynamic Capasity Networks (DCN), Jaderberg et al.”
Learning to Super-Resolve Blurry Face and Text Images (2017, background): “Deep learning achieves promising performance on many applications [21, 52, 1, 14].”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, background): “[57] introduced the dynamic capacity network (DCN) that combined two types of modules: the small subnetworks with low capacity, and the large ones with high capacity.”
Dynamic Channel and Layer Gating in Convolutional Neural Networks (2020, methodology): “Dynamic capacity networks (DCN) [1] picked-up this idea by using a high capacity and a low capacity sub-network.”