Conditional Computation in Neural Networks for faster models

By Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, Doina Precup. 2015

375 citations recorded by the source corpus.

Deep learning has become the state-of-art tool in many applications, but the evaluation and training of deep models can be time-consuming and computationally expensive. The conditional computation approach has been proposed to tackle this problem (Bengio et al., 2013; Davis & Arel, 2013). It operates by selectively activating only parts of the network at a time. In this paper, we use reinforcement learning as a tool to optimize conditional computation policies. More specifically, we cast the problem of learning activation-dependent policies for dropping out blocks of units as a reinforcement learning problem. We propose a learning scheme motivated by computation speed, capturing the idea of wanting to have parsimonious activations while maintaining prediction accuracy. We apply a policy gradient algorithm for learning policies that optimize this loss function and propose a regularization mechanism that encourages diversification of the dropout policy. We present encouraging empirical results showing that this approach improves the speed of computation without impacting the quality of the approximation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “…forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015).”
Adaptive Computation Time for Recurrent Neural Networks (2016, background): “More broadly, learning when to halt can be seen as a form of conditional computing, where parts of the network are selectively enabled and disabled according to a learned policy [3, 6].”
Unified Scaling Laws for Routed Language Models (2022, methodology): “Enter Routing Networks: models with the unusual property that each input interacts with only a subset of the network’s parameters — chosen independently for each datapoint [Bengio et al., 2016, 2013, Denoyer and Gallinari, 2014].”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “To more efficiently increase model capacity, conditional compute strategies have been developed (Bengio et al., 2013; Davis and Arel, 2013; Cho and Bengio, 2014; Bengio et al., 2015), where each input activates a subset of the model.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Further, we demonstrate how conditional computation can speed up the training [25] and how sparsely gating/routing each token through the network can efficiently be learned without any prior knowledge on task or language relatedness, exemplifying the capability of learning the routing decision direc”
A Review of Sparse Expert Models in Deep Learning (2022, neutral): “The problem of expert selection can be recast as a Bandit problem and several works have used reinforcement learning to learn the selection (Bengio et al., 2016; Rosenbaum et al., 2017; 2019; Clark et al., 2022).”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “Existing sparse gates are not differentiable, and reinforcement learning algorithms are commonly used to train these gates [1, 29].”
Dynamic Neural Networks: A Survey (2021, background): “Early studies learn to adaptively control the neuron activations by auxiliary branches [72], [73], [74] or other techniques such as low-rank approximation [75].”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “As a majority of conditional com- putation approaches make use of stochastic binary units that pose trainability challenges, Bengio et al. (2015a) cast the problem as a reinforcement learning problem and proposed a policy that maps the activations of layers to Bernoulli masks.”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Finally, our approach is also related to conditional computation methods [34,35,36,37,38,39,40,41,42,43], which learn to drop out blocks of units.”
Learning Sparse Neural Networks through L0 Regularization (2017, neutral): “This is in contrast to our approach where sparsiﬁcation happens during training, thus theoretically allowing conditional computation to speed-up training (Bengio et al., 2013; 2015).”
Learning to Continually Learn (2020, methodology): “Gating functions for such conditional computation [3] have been learned via the REINFORCE algorithm [2] and at scale via a standard back-propagation in a sparsely-gated mixture-of-experts [38], but these works focus on computational capacity and efﬁciency rather than a r X i v : 2002 .”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, background): “In this section we would like to generalize our interpretation of a large set of different weight sharing schemes including (Riemer et al., 2015; Bengio et al., 2015; Rosenbaum et al., 2018; Serrà et al., 2018) and how the concept of weight sharing impacts the dynamics of transfer (equation 1) and…”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “These include decisions trained with REINFORCE (Denoyer & Gallinari, 2014), (Bengio et al., 2015), (Hamrick et al., 2017), Q Learning (Liu & Deng, 2017), and actor-critic methods (McGill & Perona, 2017).”
Adaptive Neural Networks for Efficient Inference (2017, background): “More recently, researchers have designed spatially adaptive networks (Figurnov et al., 2016a; Bengio et al., 2015) where nodes in a layer are selectively activated.”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “To address this issue, a number of models have been introduced to enforce specialization [Jacobs et al., 1991b, Miikkulainen, 1993, Jordan and Jacobs, 1994, Bengio et al., 2013, Davis and Arel, 2013, Andreas et al., 2015, Bengio et al., 2015, Shazeer et al., 2017, Fernando et al., 2017].”
Metacontrol for Adaptive Imagination-Based Optimization (2017, background): “Our work is also related to recent notions of “conditional computation” (Bengio, 2013; Bengio et al., 2015), which adaptively modiﬁes network structure online, and “adaptive computation time” (Graves, 2016) which allows for variable numbers of internal “pondering” iterations to optimize…”
Modular Networks: Learning to Decompose Neural Computation (2018, methodology): “REINFORCE [30] maximizes the lower bound on the log likelihood L .”
Spatially Adaptive Computation Time for Residual Networks (2016, methodology): “[3] proposes to tune the amount of computation in a fully-connected network using a REINFORCE-trained policy which makes the optimization problem significantly more challenging.”