Offline Meta Reinforcement Learning
By Ron Dorfman, Aviv Tamar. 2020
21 citations recorded by the source corpus.
Consider the following problem, which we term Offline Meta Reinforcement Learning (OMRL): given the complete training histories of $N$ conventional RL agents, trained on $N$ different tasks, design a learning agent that can quickly maximize reward in a new, unseen task from the same task distribution. In particular, while each conventional RL agent explored and exploited its own different task, the OMRL agent must identify regularities in the data that lead to effective exploration/exploitation in the unseen task. To solve OMRL, we take a Bayesian RL (BRL) view, and seek to learn a Bayes-optimal policy from the offline data. We extend the recently proposed VariBAD BRL algorithm to the off-policy setting, and demonstrate learning of Bayes-optimal exploration strategies from offline data using deep neural networks. Furthermore, when applied to the online meta-RL setting (agent simultaneously collects data and improves its meta-RL policy), our method is significantly more sample efficient than the conventional VariBAD.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Chelsea Finn
- Xue Bin Peng
- Rafael Rafailov
- Pieter Abbeel
- Eric Mitchell
- Glen Berseth
- Grace Zhang
Citations in the corpus
- CoMPS: Continual Meta Policy Search (2021, methodology): “…et al., 2017; Stadie et al., 2018; Rakelly et al., 2019; Zintgraf et al., 2020; Sæmundsson et al., 2018; Fakoor et al., 2019; Sun et al., 2019; Dorfman & Tamar, 2020), applying gradient descent from initial parameters (Finn et al., 2017; Gupta et al., 2018b; Rothfuss et al., 2018; Clavera et…”
- Offline Meta-Reinforcement Learning with Advantage Weighting (2020, background): “The problem of learning to explore has largely been considered in on-policy settings in the past (Gupta et al., 2018; Zintgraf et al., 2020) but also recently in offline settings (Dorfman and Tamar, 2020).”