VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

By Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, Shimon Whiteson. 2019

331 citations recorded by the source corpus.

Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We further evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher online return than existing methods.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CoMPS: Continual Meta Policy Search (2021, methodology): “Such methods can be exceptionally powerful, learning to solve tasks that are structurally similar to the meta-training tasks with just a few dozen trials (Finn et al., 2017; Duan et al., 2016; Wang et al., 2016; Zintgraf et al., 2020).”
Offline Meta-Reinforcement Learning with Advantage Weighting (2020, methodology): “The problem of learning to explore has largely been considered in on-policy settings in the past (Gupta et al., 2018; Zintgraf et al., 2020) but also recently in offline settings (Dorfman and Tamar, 2020).”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, methodology): “We use the same network structure and training methods of VAE introduced in variBAD [56] for our latent dynamics network (Figure 2).”
Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning (2020, result): “We also compare to varibad (Zintgraf et al., 2020), which is a stable version of RL2 (Duan et al.”
Deep Reinforcement Learning amidst Continual Structured Non-Stationarity (2021, methodology): “…et al., 2019; Zintgraf et al., 2019; Stadie et al., 2018) and context-based, which includes both recurrent architectures (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2018) and architectures based on latent variable inference (Rakelly et al., 2019; Lee et al., 2019a; Zintgraf et al., 2020).”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “VariBad [Zintgraf et al., 2019]: This baseline trains a VAE to learn latent representations of the observation and reward functions, as they are perceived by the controlled agent, conditioned on the observation, action, reward triplet of the controlled agent.”
Deep Reinforcement Learning amidst Lifelong Non-Stationarity (2020, methodology): “While the DP-MDP formulation represents a strict generalization of the commonly-considered metareinforcement learning settings (typically, a BAMDP [60]), it is still somewhat limited in its generality.”
Skill-based Meta-Reinforcement Learning (2022, background): “…evaluated on tasks that span only a few dozen time steps and provide dense rewards (Finn et al., 2017; Rothfuss et al., 2019; Rakelly et al., 2019; Zintgraf et al., 2020), our tasks require to learn long-horizon behaviors over hundreds of time steps from sparse reward feedback and thus pose a new…”
Transformers are Meta-Reinforcement Learners (2022, methodology): “These results support the effectiveness of the VariBAD objective that incorporates task uncertainty directly during action selection (Zintgraf et al., 2020).”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, methodology): “Meta-RL (Duan et al., 2016; Wang et al., 2016; Finn et al., 2017a; Mishra et al., 2017; Rakelly et al., 2019; Mendonca et al., 2019; Zintgraf et al., 2020; Fakoor et al., 2020) and meta-imitation methods (Duan et al., 2017; Finn et al., 2017c; Huang et al., 2018; James et al., 2018; Paine et al.,…”
Offline Meta Reinforcement Learning (2020, methodology): “[39] showed that meta-RL is in fact an instance of BRL, where the meta-RL environment distribution is simply the BRL prior.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “reinforcement learning [107], Bayesian inference [108] and evolutionary algorithms [109] in an attempt to automatically find optimization strategies through heuristic algorithms.”
Offline Meta Reinforcement Learning - Identifiability Challenges and Effective Data Collection Strategies (2021, methodology): “Similar to the idea of solving a POMDP by representing it as an MDP over belief states, the state in BRL can be augmented with the belief to result in the Bayes-Adaptive MDP model [BAMDP, 6].”