Learning Rapid-Temporal Adaptations

By Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, Tong Wang, Adam Trischler. 2017

13 citations recorded by the source corpus.

A hallmark of human intelligence and cognition is its flexibility. One of the long-standing goals in AI research is to replicate this flexibility in a learning machine. In this work we describe a mechanism by which artificial neural networks can learn rapid-temporal adaptation - the ability to adapt quickly to new environments or tasks - that we call adaptive neurons. Adaptive neurons modify their activations with task-specific values retrieved from a working memory. On standard metalearning and few-shot learning benchmarks in both vision and language domains, models augmented with adaptive neurons achieve state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Chelsea Finn
3. Pieter Abbeel
4. Yoshua Bengio
5. Ruslan Salakhutdinov
6. Trevor Darrell
7. Mohit Bansal
8. Razvan Pascanu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Sergey Levine
5. Xiaogang Wang
6. Jitendra Malik
7. Ming-Hsuan Yang
8. Bernt Schiele

In Meta & Continual Learning

1. Chelsea Finn
2. Sergey Levine
3. Han-Jia Ye
4. Razvan Pascanu
5. Pieter Abbeel
6. Anusha Nagabandi
7. Wei-Lun Chao
8. Andrei A. Rusu

Citations in the corpus

• Meta-Learning with Latent Embedding Optimization (2018, methodology): “In the context of few-shot adaptation, additive biases across deep networks have been conditionally generated on the few-shot training samples in [28], while in [10, 31] linear output weights for novel categories were deterministically generated using the same conditioning.”
• Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, methodology): “Another popular meta-learning approach involves simply unrolling a recurrent neural network (RNN) that ingests the data (Santoro et al., 2016; Munkhdalai and Yu, 2017; Munkhdalai et al., 2017; Mishra et al., 2017) and learns internal representations of the algorithms themselves, one instantiation of”
• Meta-Learning with Warped Gradient Descent (2019, background): “Empirical evidence from gradient-based few-shot learning suggests enforcing some form of strict parameter sharing improve performance by reducing overfitting [22, 31, 33], more robust convergence [48, 38], or superior final performance [35, 42, 21].”
• MetaGAN: An Adversarial Approach to Few-Shot Learning (2018, background): “…[Sung et al., 2018, Snell et al., 2017], a shared initialization of network parameters [Finn et al., 2017], shared optimization algorithms [Ravi and Larochelle, 2017, Munkhdalai et al., 2017, Munkhdalai and Yu, 2017], or generic inference networks [Santoro et al., 2016, Mishra et al., 2018] .”
• Learning to Adapt: Meta-Learning for Model-Based Control (2018, unknown): “Another popular meta-learning approach involves simply unrolling a recurrent neural network (RNN) that ingests the data (Santoro et al., 2016; Munkhdalai & Yu, 2017; Munkhdalai et al., 2017; Mishra et al., 2017) and learns internal representations of the algorithms themselves.”