Variable Computation in Recurrent Neural Networks

By Yacine Jernite, Edouard Grave, Armand Joulin, Tomas Mikolov. 2016

60 citations recorded by the source corpus.

Recurrent neural networks (RNNs) have been used extensively and with increasing success to model various types of sequential data. Much of this progress has been achieved through devising recurrent units and architectures with the flexibility to capture complex statistics in the data, such as long range dependency or localized attention phenomena. However, while many sequential data (such as video, speech or language) can have highly variable information flow, most recurrent models still consume input features at a constant rate and perform a constant number of computations per time step, which can be detrimental to both speed and model capacity. In this paper, we explore a modification to existing recurrent units which allows them to learn to vary the amount of computation they perform at each step, without prior knowledge of the sequence's time structure. We show experimentally that not only do our models require fewer operations, they also lead to better performance overall on evaluation tasks.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Neural Ordinary Differential Equations (2018, neutral): “Adaptive computation One can adapt computation time by training secondary neural networks to choose the number of evaluations of recurrent or residual networks (Graves, 2016; Jernite et al., 2016; Figurnov et al., 2017; Chang et al., 2018).”
Reducing Transformer Depth on Demand with Structured Dropout (2019, background): “Others have proposed trainable adaptive mechanisms to control their memory footprint (Jernite et al., 2016; Sukhbaatar et al., 2019; Correia et al., 2019).”
Adaptive Attention Span in Transformers (2019, background): “This soft masking function is inspired by Jernite et al. (2017). In Figure 2, we show the shape of this piecewise function as a function of the distance.”
Recurrent Independent Mechanisms (2019, background): “Computation on demand: There are various architectures (El Hihi & Bengio, 1996; Koutnik et al., 2014; Chung et al., 2016; Neil et al., 2016; Jernite et al., 2016; Krueger et al., 2016) where parts of the LSTM’s hidden state are kept dormant at times.”
Learning to Skim Text (2017, methodology): “Our method belongs to adaptive computation of neural networks, whose idea is recently explored by (Graves, 2016; Jernite et al., 2016), where different amount of computations are allocated dynamically per time step.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, methodology): “To avoid the difﬁculties of reinforcement learning, often a soft approximation to the hard selection decision is used during training [11, 35, 36, 37, 38, 39, 40, 41, 42].”
Neural Speed Reading via Skim-RNN (2017, methodology): “Variable Computation in RNN (VCRNN) (Jernite et al., 2017) is also concerned with dynamically controlling the computational cost of RNN.”
Dynamic Computational Time for Visual Attention (2017, background): “[41] learn a scheduler to determine what portion of the hidden state to compute based on the current hidden and input vectors.”
Recurrent Neural Networks (2018, background): “Other works have addressed time-dependent computation in RNNs by updating only a fraction of the hidden states based on the current hidden state and input (Jernite et al., 2017), or following periodic patterns (Koutnik et al.”
Confident Adaptive Language Modeling (2022, background): “Another line of work involves conditional computation to train larger models that only use a sparser subset of the full network during inference, for example by routing over mixture-of-experts [9; 22; 39; 91], recurring modules [18; 29; 35], or accessing external memory [82].”
Memory Architectures in Recurrent Neural Network Language Models (2018, background): “As a result, our stack is also related to variable computation recurrent networks Jernite et al. (2017) that decide the number of dimensions to be used at each timestep.”
Explicitly Modeling Adaptive Depths for Transformer (2020, background): “Another group of works explore to conduct conditional computation inside the dimension of neural networks, (Jernite et al., 2017; Shen et al., 2019), instead of activating partial layers of model, e.g., adaptive depths in our method.”
On the Transformer Growth for Progressive BERT Training (2020, background): “, 2017) and Adaptive Computation Time (Graves, 2016; Jernite et al., 2016).”
Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “To this end, a masking function inspired by [111] is used to control the attention span for each head.”