Meta-learning of Sequential Strategies

By Pedro A. Ortega, Jane X. Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Manfred Otto Heess, Joel Veness, Alexander Pritzel, Pablo Sprechmann and 14 more. 2019

120 citations recorded by the source corpus.

In this report we review memory-based meta-learning as a tool for building sample-efficient strategies that learn from past experience to adapt to any task within a target class. Our goal is to equip the reader with the conceptual foundations of this tool for building new, scalable agents that operate on broad domains. To do so, we present basic algorithmic templates for building near-optimal predictors and reinforcement learners which behave as if they had a probabilistic model that allowed them to efficiently exploit task structure. Furthermore, we recast memory-based meta-learning within a Bayesian framework, showing that the meta-learned strategies are near-optimal because they amortize Bayes-filtered data, where the adaptation is implemented in the memory dynamics as a state-machine of sufficient statistics. Essentially, memory-based meta-learning translates the hard problem of probabilistic sequential inference into a regression problem.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Offline Meta Reinforcement Learning - Identifiability Challenges and Effective Data Collection Strategies (2021, background): “The connection between meta-learning and Bayesian methods, and between meta-RL and Bayesian RL in particular, has been investigated in a series of recent papers [20, 18, 26, 36], and our work closely follows these ideas.”
Transformers are Meta-Reinforcement Learners (2022, background): “Another branch of methods learns the entire system using memory-based architectures (Ortega et al., 2019; Wang et al., 2017; Duan et al., 2016; Ritter et al., 2018a) or generating update rules by discovery (Oh et al., 2020) or evolution (Co-Reyes et al., 2021).”
Transformers Can Do Bayesian Inference (2021, background): “One more related line of research is amortized simulation-based inference (Cranmer et al., 2020; Chan et al., 2018), which was previously adapted to general probabilistic programs (Ritchie et al., 2016) and sequences (Ortega et al., 2019).”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, unknown): “Meta-learning has been both combined with imitation learning [18, 25, 83, 34, 8, 52, 84] and reinforcement learning through context space learning [79, 17, 47, 59, 90] and gradient-based optimization [24, 61, 33, 28, 46].”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Recent works have demonstrated the effectiveness of agent-environment co-adaptation with a distribution of tasks (Wang et al., 2019; Akkaya et al., 2019; Wang et al., 2020a; Parker-Holder et al., 2022).”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, background): “oder to reconstruct previous and future transitions / rewards (which acts as an auxiliary loss (Jaderberg et al., 2017) to encode the task in latent space and deduce information about unseen states). Ortega et al. (2019) provide an in-depth discussion of meta-learning sequential strategies and how t”
Bayesian Meta Sampling for Fast Uncertainty Adaptation (2020, background): “…(Nichol et al., 2018), and the lately extensions to an online setting (Finn et al., 2019), to model hierarchical relation (Yao et al., 2019) and sequential strategies (Ortega et al., 2019), and to its stable version Antoniou et al. (2019) and to some theoretical analysis (Khodak et al., 2019).”
Offline Meta Reinforcement Learning (2020, background): “The connection between meta-learning and Bayesian methods, and between meta-RL and Bayesian RL in particular, has been investigated in a series of recent papers [24, 22, 39, 28], and our work closely follows these ideas.”