Changing Model Behavior at Test-Time Using Reinforcement Learning

By Augustus Odena, Dieterich Lawson, Christopher Olah. 2017

57 citations recorded by the source corpus.

Machine learning models are often used at test-time subject to constraints and trade-offs not present at training-time. For example, a computer vision model operating on an embedded device may need to perform real-time inference, or a translation model operating on a cell phone may wish to bound its average compute time in order to be power-efficient. In this work we describe a mixture-of-experts model and show how to change its test-time resource-usage on a per-input basis using reinforcement learning. We test our method on a small MNIST-based example.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Adaptive Neural Networks for Efficient Inference (2017, background): “Others have developed cascade approaches (Leroux et al., 2017; Odena et al., 2017) that allow early exits based on conﬁ-dence feedback.”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “A number of approaches [29, 34, 49, 50] proposed to train a separate network via RL to decide when to halt.”
Dynamic Channel Pruning: Feature Boosting and Suppression (2018, methodology): “Odena et al. (2017) designed a DNN with layers containing multiple modules, and decided which module to use with a recurrent neural network (RNN).”
Learning Instance-wise Sparsity for Accelerating Deep Models (2019, methodology): “For example, [Gao et al., 2018; Lin et al., 2017; Odena et al., 2017; Hua et al., 2018] utilized attention and gate layers to evaluate each channel and discard some of them with subtle importances during the inference phrase.”
Dynamic Computational Time for Visual Attention (2017, background): “[42] is the first attempt that learn to change model behavior at test time with reinforcement learning.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “This often takes the form of a mixture-of-experts [23, 24], in which a learned controller is used to select the experts relevant for a given input [25, 26, 27, 28, 29, 30], possibly hierarchically [31, 32, 33, 34].”
Multi-Scale Dense Networks for Resource Efficient Image Classification (2017, background): “In (Odena et al., 2017), a “composer”model is trained to construct the evaluation network from a set of sub-modules for each test example.”
Fully Dynamic Inference With Deep Neural Networks (2020, methodology): “propose adaptively constructing computation graphs from submodules using a reinforcement learning-based controller [32].”
Learning Layer-Skippable Inference Network (2020, background): “[40] leveraged a Recurrent Neural Network (RNN) controller to”
Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks (2017, background): “Other approaches are dynamic networks which conditionally select the modules to respect a budget objective.[2, 22, 14, 1, 18].”
Runtime Neural Pruning (2017, background): “Dynamic network: Dynamic network structures and executions have been studied in previous works [7, 28, 33, 39, 40].”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Stopping criteria: a) in the case of layer-wise and networkwise pruning, some pruning algorithms skip the pruned layer/network [19, 237], b) some algorithms dynamically choose the data path [179, 249], and c) while the others end the computation and output the predicting results [64, 136, 139].”
Neural Speed Reading via Skim-RNN (2017, background): “cupies less memory. Rastegari et al. (2016) approximate 32-bit ﬂoat operations with single bit binary operations to substantially increase computational speed at the cost of little loss of precision. Odena et al. (2017) propose to change model behavior on per-input basis, which can decide to use les”