Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning

By Kyunghyun Cho, Yoshua Bengio. 2014

44 citations recorded by the source corpus.

Many state-of-the-art results obtained with deep networks are achieved with the largest models that could be trained, and if more computation power was available, we might be able to exploit much larger datasets in order to improve generalization ability. Whereas in learning algorithms such as decision trees the ratio of capacity (e.g., the number of parameters) to computation is very favorable (up to exponentially more parameters than computation), the ratio is essentially 1 for deep neural networks. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation required, by activating some parameters and computation "on-demand", on a per-example basis. In this note, we propose a novel parametrization of weight matrices in neural networks which has the potential to increase up to exponentially the ratio of the number of parameters to computation. The proposed approach is based on turning on some parameters (weight matrices) when specific bit patterns of hidden unit activations are obtained. In order to better control for the overfitting that might result, we propose a parametrization that is tree-structured, where each node of the tree corresponds to a prefix of a sequence of sign bits, or gating units, associated with hidden units.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “A MoE whose experts are simple weight matrices is similar to the parameterized weight matrix proposed in (Cho & Bengio, 2014).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “To more efficiently increase model capacity, conditional compute strategies have been developed (Bengio et al., 2013; Davis and Arel, 2013; Cho and Bengio, 2014; Bengio et al., 2015), where each input activates a subset of the model.”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, methodology): “Gating techniques relying on sampling and straight-through gradient estimators are common (Bengio et al., 2013; Eigen et al., 2013; Cho and Bengio, 2014).”
Mixture-of-Experts with Expert Choice Routing (2022, background): “Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation, by activating certain parameters and computation on demand, on a per-example or per-token basis [3].”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, methodology): “To facilitate such routing, different approaches have been developed, including large mixture of experts (Shazeer et al., 2017) or gating (Eigen et al., 2013; Cho and Bengio, 2014).”
Large Memory Layers with Product Keys (2019, methodology): “Neural networks are commonly employed to address many complex tasks such as machine translation [6], image classification [26] or speech recognition [15].”
Dynamic Neural Networks: A Survey (2021, background): “Early studies learn to adaptively control the neuron activations by auxiliary branches [72], [73], [74] or other techniques such as low-rank approximation [75].”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “Following this line of research, Cho & Bengio (2014) studied increasing the capacity of neural networks without increasing required computation by exploiting the bit patterns associated with hidden units.”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Finally, our approach is also related to conditional computation methods [34,35,36,37,38,39,40,41,42,43], which learn to drop out blocks of units.”
Scaling Vision with Sparse Mixture of Experts (2021, background): “To grow the number of model parameters without proportionally increasing the computational cost, conditional computation [5, 15, 12] only activates some relevant parts of the model in an input-dependent fashion, like in decision trees [7].”
Deep Mixture of Experts via Shallow Embedding (2018, background): “[2, 3, 4] explored conditional computation in the context of neural networks which selectively executes part of the network based on the input.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “This often takes the form of a mixture-of-experts [23, 24], in which a learned controller is used to select the experts relevant for a given input [25, 26, 27, 28, 29, 30], possibly hierarchically [31, 32, 33, 34].”
CondConv: Conditionally Parameterized Convolutions for Efficient Inference (2019, background): “In conditional computation models, this is achieved by activating only a portion of the entire network for each example [3, 8, 5, 2].”
Conditional Computation for Continual Learning (2019, background): “1 Introduction In this paper, we study the level of parameter sharing within the conditional computation framework [2, 3, 1, 10].”
Zoo-Tuning: Adaptive Transfer from a Zoo of Models (2021, methodology): “Our method is also related to conditional computation (Davis & Arel, 2013; Cho & Bengio, 2014), where parts of the network are active on a perexample basis.”