Controlling Computation versus Quality for Neural Sequence Models

By Ankur Bapna, Naveen Arivazhagan, Orhan Firat. 2020

35 citations recorded by the source corpus.

Most neural networks utilize the same amount of compute for every example independent of the inherent complexity of the input. Further, methods that adapt the amount of computation to the example focus on finding a fixed inference-time computational graph per example, ignoring any external computational budgets or varying inference time limitations. In this work, we utilize conditional computation to make neural sequence models (Transformer) more efficient and computation-aware during inference. We first modify the Transformer architecture, making each set of operations conditionally executable depending on the output of a learned control network. We then train this model in a multi-task setting, where each task corresponds to a particular computation budget. This allows us to train a single model that can be controlled to operate on different points of the computation-quality trade-off curve, depending on the available computation budget at inference time. We evaluate our approach on two tasks: (i) WMT English-French Translation and (ii) Unsupervised representation learning (BERT). Our experiments demonstrate that the proposed Conditional Computation Transformer (CCT) is competitive with vanilla Transformers when allowed to utilize its full computational budget, while improving significantly over computationally equivalent baselines when operating on smaller computational budgets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Conditional computation [25, 16, 26, 27] enables us to satisfy training and inference efficiency by having a sub-network activated on the per-input basis.”
Fast Inference from Transformers via Speculative Decoding (2022, methodology): “These adaptive computation methods (e.g. Han et al., 2021; Sukhbaatar et al., 2019; Schuster et al., 2021; Scardapane et al., 2020; Bapna et al., 2020; Elbayad et al., 2019; Schwartz et al., 2020) aim to use less compute resources for easier inference steps.”
Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (2021, methodology): “…et al. (2020a) applied conditional computation to sparsify encoder outputs in sequence-to-sequence models in order to reduce attention costs, while Bapna et al. (2020) introduced conditional execution of Transformer sub-layers to control the amount of computation expended by the model at…”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, methodology): “To avoid the difﬁculties of reinforcement learning, often a soft approximation to the hard selection decision is used during training [11, 35, 36, 37, 38, 39, 40, 41, 42].”
A Survey of Transformers (2021, methodology): “ACT UT[26], Conditional Computation Transformer[7], DeeBERT[150], PABEE[171], Li et al.”
Confident Adaptive Language Modeling (2022, background): “Another line of work involves conditional computation to train larger models that only use a sparser subset of the full network during inference, for example by routing over mixture-ofexperts [9; 22; 39; 91], recurring modules [18; 29; 35], or accessing external memory [82].”