Surprisal-Triggered Conditional Computation with Neural Networks
By Loren Lugosch, Derek Nowrouzezahrai, B. Meyer. 2020
6 citations recorded by the source corpus.
Autoregressive neural network models have been used successfully for sequence generation, feature extraction, and hypothesis scoring. This paper presents yet another use for these models: allocating more computation to more difficult inputs. In our model, an autoregressive model is used both to extract features and to predict observations in a stream of input observations. The surprisal of the input, measured as the negative log-likelihood of the current observation according to the autoregressive model, is used as a measure of input difficulty. This in turn determines whether a small, fast network, or a big, slow network, is used. Experiments on two speech recognition tasks show that our model can match the performance of a baseline in which the big network is always used with 15% fewer FLOPs.
Subfields: Artificial Intelligence, Efficiency & Compression, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Orhan Firat
- Yanping Huang
- Graham Neubig
- Mohit Bansal
- Kyunghyun Cho
- Luke S. Zettlemoyer
- Jianfeng Gao
In Efficiency & Compression
- Yanping Huang
- Orhan Firat
- Maxim Krikun
- Dmitry Lepikhin
- Z. Chen
- Yuanzhong Xu
- Charlie Nash
- Noam Shazeer
In Signal Processing
- Jesse Engel
- Sander Dieleman
- Karen Simonyan
- Charlie Nash
- Noam Shazeer
- Z. Chen
- Orhan Firat
- Junichi Yamagishi
Citations in the corpus
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “The routing strategy can be any of the following: estimated difficulty of the example (Lugosch et al., 2020), available computation budget (Elbayad et al., 2020; Bapna et al., 2020), or more generally a learned criterion with sparsity induced mixture of experts (Shazeer et al., 2017).”
- Variable-rate discrete representation learning (2021, background): “Other models operate on fixed-quantum inputs, but adaptively allocate capacity 5 (Eslami et al., 2016; Graves, 2016; Yeung et al., 2016; Campos et al., 2017; Fig-urnov et al., 2017; Malinowski et al., 2018; Marin et al., 2019; Kirillov et al., 2020; Lugosch et al., 2020).”