Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks

By Andrew S. Davis, Itamar Arel. 2013

88 citations recorded by the source corpus.

Scalability properties of deep neural networks raise key research questions, particularly as the problems considered become larger and more challenging. This paper expands on the idea of conditional computation introduced by Bengio, et. al., where the nodes of a deep network are augmented by a set of gating units that determine when a node should be calculated. By factorizing the weight matrix into a low-rank approximation, an estimation of the sign of the pre-nonlinearity activation can be efficiently obtained. For networks using rectified-linear hidden units, this implies that the computation of a hidden unit with an estimated negative pre-nonlinearity can be ommitted altogether, as its value will become zero when nonlinearity is applied. For sparse neural networks, this can result in considerable speed gains. Experimental results using the MNIST and SVHN data sets with a fully-connected deep neural network demonstrate the performance robustness of the proposed scheme with respect to the error introduced by the conditional computation process.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “…forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “To more efficiently increase model capacity, conditional compute strategies have been developed (Bengio et al., 2013; Davis and Arel, 2013; Cho and Bengio, 2014; Bengio et al., 2015), where each input activates a subset of the model.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “In this section, we advocate how conditional computation (Bengio et al., 2013; Davis & Arel, 2013) with sparsely gated mixture of experts ﬁts into the above detailed desiderata and show its efﬁcacy by scaling neural machine translation models, while keeping the training time of such massive…”
Dynamic Neural Networks: A Survey (2021, background): “Early studies learn to adaptively control the neuron activations by auxiliary branches [72], [73], [74] or other techniques such as low-rank approximation [75].”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “Activating a sub-network depending on the particular input example has been the focus of conditional computation approaches (Bengio et al., 2013; Davis & Arel, 2014).”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Finally, our approach is also related to conditional computation methods [34,35,36,37,38,39,40,41,42,43], which learn to drop out blocks of units.”
Conditional Computation in Neural Networks for faster models (2015, background): “Recent approaches (Bengio et al., 2013; Davis & Arel, 2013) have proposed the use of conditional computation in order to address this problem.”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “To address this issue, a number of models have been introduced to enforce specialization [Jacobs et al., 1991b, Miikkulainen, 1993, Jordan and Jacobs, 1994, Bengio et al., 2013, Davis and Arel, 2013, Andreas et al., 2015, Bengio et al., 2015, Shazeer et al., 2017, Fernando et al., 2017].”
Variance Reduction in SGD by Distributed Importance Sampling (2015, methodology): “SGD (Davis & Arel, 2013) 9.31 SGD (ours) 0.0754 Importance Sampling SGD 0.0756 Table 1: Test Error on Permutation Invariant SVHN Datasets with different methods.”
Scaling Vision with Sparse Mixture of Experts (2021, background): “To grow the number of model parameters without proportionally increasing the computational cost, conditional computation [5, 15, 12] only activates some relevant parts of the model in an input-dependent fashion, like in decision trees [7].”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Conditional computing applies to training and inference [20, 53].”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “1) Sparse: The sparse gating functions activate a selected subset of experts for processing each individual input token, which can be considered as a form of conditional computation [134], [135], [136].”
CondConv: Conditionally Parameterized Convolutions for Efficient Inference (2019, background): “In conditional computation models, this is achieved by activating only a portion of the entire network for each example [3, 8, 5, 2].”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, background): “Sequence shortening techniques, which can be seen as a sort of conditional computation [7, 6, 15] in time, can alleviate these issues.”
Adaptive Neural Trees (2018, unknown): “Rationales for using conditional computation ranges from attaining better capacity-to-computation ratio (Bengio et al., 2013; Davis & Arel, 2013; Bengio et al., 2015; Shazeer et al., 2017) to adapting the required computation to the difﬁculty of the input and task (Bengio et al., 2015; Almahairi et…”
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs (2018, methodology): “On the other hand, Davis and Arel have used low-rank approximations to predict the sparse activations of neurons at each layer [6].”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, methodology): “Our method is also related to conditional computation (Davis & Arel, 2013; Cho & Bengio, 2014), where parts of the network are active on a perexample basis.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “In the last few years, researchers have begun thinking more about how to incorporate conditional computation into neural networks [11, 12].”