Been There, Done That: Meta-Learning with Episodic Recall

By Samuel Ritter, Jane X. Wang, Zeb Kurth-Nelson, Siddhant M. Jayakumar, Charles Blundell, Razvan Pascanu, Matthew Botvinick. 2018

95 citations recorded by the source corpus.

Meta-learning agents excel at rapidly learning new tasks from open-ended task distributions; yet, they forget what they learn about each task as soon as the next begins. When tasks reoccur - as they do in natural environments - metalearning agents must explore again instead of immediately exploiting previously discovered solutions. We propose a formalism for generating open-ended yet repetitious environments, then develop a meta-learning architecture for solving these environments. This architecture melds the standard LSTM working memory with a differentiable neural episodic memory. We explore the capabilities of agents with this episodic LSTM in five meta-learning environments with reoccurring tasks, ranging from bandits to navigation and stochastic sequential decision problems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Continual Lifelong Learning in Natural Language Processing: A Survey (2020, methodology): “As meta-learning is able to efficiently learn with limited samples, hence such a CL model would adapt quicker in dynamic environments (e.g. Ritter et al., 2018; Al-Shedivat et al., 2018).”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, background): “Other meta-learning works have considered non-stationarity within a task (Al-Shedivat et al., 2017) and episodes involving multiple tasks at meta-test time (Ritter et al., 2018), but they do not consider continual online adaptation with unknown task separation.”
Meta-Learning in Neural Networks: A Survey (2020, background): “Nevertheless, meta-learning provides a potential framework to advance continual learning, and a few recent studies have begun to do so by developing meta-objectives that encode continual learning performance [62], [63], [64].”
Learning where to learn: Gradient sparsity in meta and continual learning (2021, background): “2 We finally consider another online learning setting in which the underlying task is concealed from the learner and can randomly change at each step, potentially going back to previously seen tasks [48, 18, 7].”
Synthetic Returns for Long-Term Credit Assignment (2021, background): “This data structure has a long history in deep-RL, and is sometimes called “episodic memory” (Oh et al., 2016; Wayne et al., 2018; Ritter et al., 2018; Fortunato et al., 2019).”
Generalization of Reinforcement Learners with Working and Episodic Memory (2019, background): “More recently, Ritter et al. (2018) shows how episodic memory can be used to provide agents with context-switching abilities in contextual bandit problems.”
Rapid Task-Solving in Novel Environments (2020, background): “Ritter et al. (2018a) showed that a deep-RL agent with episodic memory and a minimal learned planning module (an MLP) could learn to produce behavior consistent with EMBC (Vikbladh et al., 2017).”
Revisiting Meta-Learning as Supervised Learning (2020, unknown): “Some algorithms aim for versatile purposes (Mishra et al., 2018; Munkhdalai and Yu, 2017; Ritter et al., 2018; Finn et al., 2017a; Santoro et al., 2016; Nichol et al., 2018).”
MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient Electronic Health Records (2019, background): “Meta-learning algorithms have been recently explored on a series of topics including few-shot learning [32, 41], reinforcement learning [15, 33] and imitation learning [16].”
A CCELERATING O NLINE R EINFORCEMENT L EARNING VIA M ODEL -B ASED M ETA -L EARNING (2021, methodology): “Meta-reinforement learning methods have also considered non-stationarity within a short task (Al-Shedivat et al., 2018) and episodes involving multiple tasks at meta-test time (Ritter et al., 2018), but they do not consider continual online adaptation with unknown task boundaries.”
Concurrent Meta Reinforcement Learning (2019, unknown): “A related work in the non-meta-learning setting is that of Concurrent Reinforcement Learning Silver et al. (2013). This work proposed a new concurrent setting beyond the typical sequential or continual settings used in most reinforcement learning formalizations. The concurrent setting is defined as ”
Meta reinforcement learning as task inference (2019, background): “To study the emergence of efﬁcient learning grounded in prior knowledge about a task distribution, several recent papers have turned to a well-established “meta” perspective on reinforcement learning (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2018; Ritter et al., 2018).”
How to Train Your MAML to Excel in Few-Shot Classification (2021, background): “Recent years have witnessed an abundance of new approaches on meta-learning [29, 30, 48, 61, 63, 64, 68, 77, 79, 80, 94, 98], developed in various areas including few-shot learning [70, 85, 89, 105, 107, 110, 111, 115], optimization [3, 9, 53, 108], reinforcement and imitation learning [23, 24, 31, ”
Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation (2020, unknown): “Recent advances [4, 17] perform meta-learning on training environments to learn how to adapt efficiently to testing environments.”
Reinforcement Learning, Fast and Slow. (2019, unknown): “Indeed, recent work has explored an approach to integrating meta-learning and episodic control, capitalizing on their complementary benefits [39, 40].”