Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
By Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, Sergey Levine. 2019
832 citations recorded by the source corpus.
Deep reinforcement learning algorithms require large amounts of experience to learn an individual task. While in principle meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience, several major challenges preclude their practicality. Current methods rely heavily on on-policy experience, limiting their sample efficiency. The also lack mechanisms to reason about task uncertainty when adapting to new tasks, limiting their effectiveness in sparse reward problems. In this paper, we address these challenges by developing an off-policy meta-RL algorithm that disentangles task inference and control. In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience. This probabilistic interpretation enables posterior sampling for structured and efficient exploration. We demonstrate how to integrate these task variables with off-policy RL algorithms to achieve both meta-training and adaptation efficiency. Our method outperforms prior algorithms in sample efficiency by 20-100X as well as in asymptotic performance on several meta-RL benchmarks.
Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Chelsea Finn
- Yoshua Bengio
- Pieter Abbeel
- Ruslan Salakhutdinov
- Mohit Bansal
- Dacheng Tao
- Quoc V. Le
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhishek Gupta
- Shimon Whiteson
- Ofir Nachum
- Shixiang Shane Gu
- George Tucker
In Meta & Continual Learning
- Chelsea Finn
- Sergey Levine
- Han-Jia Ye
- Dacheng Tao
- James Harrison
- Annie Xie
- Abhishek Gupta
- Pieter Abbeel
Citations in the corpus
- Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “For instance, context-based meta-RL methods adapt to new tasks by aggregating experience into a latent representation on which the policy is conditioned [8, 9, 10, 11].”
- CoMPS: Continual Meta Policy Search (2021, methodology): “Reward function: We modified the reward function in PEARL Rakelly et al. (2019), reducing the control cost portion of the reward by an order of magnitude.”
- Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, background): “learning [5, 42, 24, 30], extending contextual meta-learning approaches.”
- Meta-Learning in Neural Networks: A Survey (2020, background): “…[91] [41], [75], [103], [160], [161] [77], [89] Hyperparam [25] [162], [163] [159] [164] Black-box model [40], [82], [165], [166] [22], [106], [109] Metric [20], [86], [87] Loss/Reward [44], [92] Exploration Policy [24], [168]–[172] Dataset/Environment [144] [146] [149] [150] Instance Weights…”
- Meta-Learning with Warped Gradient Descent (2019, background): “Empirical evidence from gradient-based few-shot learning suggests enforcing some form of strict parameter sharing improve performance by reducing overfitting [22, 31, 33], more robust convergence [48, 38], or superior final performance [35, 42, 21].”
- Block Contextual MDPs for Continual Learning (2021, methodology): “Meta-reinforcement learning methods (Finn et al., 2017; Nagabandi et al., 2019a; Rakelly et al., 2019; Zhao et al., 2020) can be broadly classified as:i) Optimization-based methods (Finn et al.”
- Human-Timescale Adaptation in an Open-Ended Task Space (2023, neutral): “, 2021), and latent-variable based approaches (Finn et al., 2018; Humplik et al., 2019; Rakelly et al., 2019; Zintgraf et al., 2019).”
- Deep Reinforcement Learning amidst Continual Structured Non-Stationarity (2021, methodology): “…et al., 2019; Zintgraf et al., 2019; Stadie et al., 2018) and context-based, which includes both recurrent architectures (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2018) and architectures based on latent variable inference (Rakelly et al., 2019; Lee et al., 2019a; Zintgraf et al., 2020).”
- Fast Adaptive Task Offloading in Edge Computing Based on Meta Reinforcement Learning (2020, neutral): “possible solution could be to integrate the seq2seq neural network with another sample efficient off-policy MRL method [28], which is a direction for future work.”
- Model-based Adversarial Meta-Reinforcement Learning (2020, methodology): “Following and extending the setup of [13, 38], we first consider a family of environments and tasks relating to controlling 2-D or 3-D velocity control tasks.”
- Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “These approaches have been extended into the setting of RL (Duan et al., 2016b; Wang et al., 2016; Finn et al., 2017a; Sung et al., 2017; Gupta et al., 2018; Men-donca et al., 2019; Houthooft et al., 2018; Stadie et al., 2018; Rakelly et al., 2019; Nagabandi et al., 2018a).”
- Guided Meta-Policy Search (2019, methodology): “Recent work has used amortized probabilistic inference [34] to achieve data-efficient meta-training, however such contextual methods cannot continually adapt to out of distribution test tasks.”
- Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, neutral): “…for fast adaptation to new environments is meta reinforcement learning (meta RL) (Cully et al., 2015; Finn et al., 2017; Wang et al., 2017; Duan et al., 2016; Xu et al., 2018; Houthooft et al., 2018; Sæmundsson et al., 2018; Nagabandi et al., 2018; Humplik et al., 2019; Rakelly et al., 2019).”