Adaptive Computation Time for Recurrent Neural Networks

By Alex Graves. 2016

785 citations recorded by the source corpus.

This paper introduces Adaptive Computation Time (ACT), an algorithm that allows recurrent neural networks to learn how many computational steps to take between receiving an input and emitting an output. ACT requires minimal changes to the network architecture, is deterministic and differentiable, and does not add any noise to the parameter gradients. Experimental results are provided for four synthetic problems: determining the parity of binary vectors, applying binary logic operations, adding integers, and sorting real numbers. Overall, performance is dramatically improved by the use of ACT, which successfully adapts the number of computational steps to the requirements of the problem. We also present character-level language modelling results on the Hutter prize Wikipedia dataset. In this case ACT does not yield large gains in performance; however it does provide intriguing insight into the structure of the data, with more computation allocated to harder-to-predict transitions, such as spaces between words and ends of sentences. This suggests that ACT or other adaptive computation methods could provide a generic method for inferring segment boundaries in sequence data.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Universal Transformers (2018, methodology): “Adaptive Computation Time (ACT) (Graves, 2016) is a mechanism for dynamically modulating the number of computational steps needed to process each input symbol 1Note that T here denotes time-step T and not the transpose operation.”
Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, positive): “1 Prior work (Graves, 2016; Banino et al., 2021) has explored neural architectures that explicitly allow for dynamically chosen amounts of computation time to be dedicated to different sub-tasks.”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “…costs of machine learning models tend to scale with sequence length, much previous work on character- and bytelevel models has explored ways to process long sequences efficiently using convolutions with pooling (Zhang et al., 2015; Lee et al., 2017) or adaptive computation time (Graves, 2016).”
Emergent Abilities of Large Language Models (2022, neutral): “Other directions for better computational efﬁciency could involve variable amounts of compute for different inputs (Graves, 2016; Dehghani et al., 2018), using more localized learning strategies than backpropagation through all weights in a neural network (Jaderberg et al., 2017), and augmenting…”
Neural Ordinary Differential Equations (2018, neutral): “Adaptive computation One can adapt computation time by training secondary neural networks to choose the number of evaluations of recurrent or residual networks (Graves, 2016; Jernite et al., 2016; Figurnov et al., 2017; Chang et al., 2018).”
Faithful Reasoning Using Large Language Models (2022, background): “Other examples in the deep learning literature draw random variables from a parameterised distribution to predict when to stop reasoning (Banino et al., 2021; Graves, 2016).”
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, methodology): “In this respect, it shares the same spirit as the methods in (Graves, 2016; Dehghani et al., 2019; Dong et al., 2019; Schwarzschild et al., 2021; Wang et al., 2021; Wei et al., 2022; Wang et al., 2022).”
Analysing Mathematical Reasoning Abilities of Neural Models (2019, critical): “We also experimented with Adaptive Computation Time as introduced in (Graves, 2016), although this yielded worse results than simply having a fixed number of “thinking” steps.”
Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “Adaptive approaches adjust the amount of computation per example to amortize the total inference cost (see Teerapittayanon et al., 2017; Graves, 2017; Huang et al., 2018; Kaya et al., 2019; Wang et al., 2018, inter alia ).”
Dynamic Neural Networks: A Survey (2021, background): “a) Halting score is first proposed in [11], where an accumulated scalar named halting score adaptively decides whether the hidden state of an RNN will be directly fed”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, methodology): “Some works introduced methods to allocate fewer resources to certain parts of the input (e.g., certain words), thereby potentially reducing training and/or inference time (Graves, 2016; Seo et al., 2018).”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “Graves (2016) proposed the first application of conditional computation to neural sequence models, called Adaptive Computation Time (ACT), where a recurrent neural network is trained to learn the lag between reading an input and generating the output.”
Depth-Adaptive Transformer (2019, positive): “We extend Graves (2016; ACT) who introduced dynamic computation to recurrent neural networks in several ways: we apply different layers at each stage, we investigate a range of designs and training targets for the halting module and we explicitly supervise through simple oracles to achieve good…”
Adaptive Attention Span in Transformers (2019, methodology): “As an extension, we consider a dynamic computation approach (Graves, 2016) where the attention span dynamically change based on the current input (Luong et al.”
The Predictron: End-To-End Learning and Planning (2016, background): “Graves (2016) introduced an architecture with adaptive computation time (ACT), with a discrete (but differentiable) decision on when to halt, and aggregating the outputs at each pondering step.”
Scalable Adaptive Computation for Iterative Generation (2022, background): “Other approaches for adaptive computation have mainly explored models with dynamic depth with recurrent networks (Graves, 2016; Figurnov et al., 2017) or sparse computation (Yin et al.”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, background): “…thereof (Peng et al., 2015; Weissenborn, 2016; Xiong et al., 2016; Liu and Perez, 2017), which often is iterated over several times (Sordoni et al., 2016; Mark et al., 2016; Seo et al., 2016; Hu et al., 2017) and may contain train-able stopping mechanisms (Graves, 2016; Shen et al., 2017b).”