PonderNet: Learning to Ponder
By Andrea Banino, Jan Balaguer, Charles Blundell. 2021
148 citations recorded by the source corpus.
In standard neural networks the amount of computation used grows with the size of the inputs, but not with the complexity of the problem being learnt. To overcome this limitation we introduce PonderNet, a new algorithm that learns to adapt the amount of computation based on the complexity of the problem at hand. PonderNet learns end-to-end the number of computational steps to achieve an effective compromise between training prediction accuracy, computational cost and generalization. On a complex synthetic problem, PonderNet dramatically improves performance over previous adaptive computation methods and additionally succeeds at extrapolation tests where traditional neural networks fail. Also, our method matched the current state of the art results on a real world question and answering dataset, but using less compute. Finally, PonderNet reached state of the art results on a complex task designed to test the reasoning capabilities of neural networks.1
Subfields: Artificial Intelligence, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jan Kautz
- Arash Vahdat
- José Manuel Álvarez
- Murray Shanahan
- Hongxu Yin
- Pavlo Molchanov
- Augustus Odena
- David Bieber
In Model Architectures
- D. Luan
- Murray Shanahan
- Hongxu Yin
- Jan Kautz
- Guy Gur-Ari
- Augustus Odena
- Pavlo Molchanov
- José Manuel Álvarez
Citations in the corpus
- Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, positive): “1 Prior work (Graves, 2016; Banino et al., 2021) has explored neural architectures that explicitly allow for dynamically chosen amounts of computation time to be dedicated to different sub-tasks.”
- Faithful Reasoning Using Large Language Models (2022, background): “Other examples in the deep learning literature draw random variables from a parameterised distribution to predict when to stop reasoning (Banino et al., 2021; Graves, 2016).”
- A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, methodology): “Similar prior distribution has been recently shown effective to stablize convergence during stochastic pondering [1].”