Guided Meta-Policy Search

By Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, Chelsea Finn. 2019

59 citations recorded by the source corpus.

Reinforcement learning (RL) algorithms have demonstrated promising results on complex tasks, yet often require impractical numbers of samples because they learn from scratch. Meta-RL aims to address this challenge by leveraging experience from previous tasks in order to more quickly solve new tasks. However, in practice, these algorithms generally also require large amounts of on-policy experience during the meta-training process, making them impractical for use in many problems. To this end, we propose to learn a reinforcement learning procedure through imitation of expert policies that solve previously-seen tasks. This involves a nested optimization, with RL in the inner loop and supervised imitation learning in the outer loop. Because the outer loop imitation learning can be done with off-policy data, we can achieve significant gains in meta-learning sample efficiency. In this paper, we show how this general idea can be used both for meta-reinforcement learning and for learning fast RL procedures from multi-task demonstration data. The former results in an approach that can leverage policies learned for previous tasks without significant amounts of on-policy data during meta-training, whereas the latter is particularly useful in cases where demonstrations are easy for a person to provide. Across a number of continuous control meta-RL problems, we demonstrate significant improvements in meta-RL sample efficiency in comparison to prior work as well as the ability to scale to domains with visual observations.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CoMPS: Continual Meta Policy Search (2021, methodology): “To reduce the cost of needing additional trajectories for the outer objective, Mendonca et al. (2019) propose meta-training with the expected discounted return as the inner task loss and supervised imitation as the outer loss: min θ ∑ Ti ∑ T vi ∼D∗0:i ETi [LBC(θ+α∇θJi(θ),Dvi )], LBC(θi,Di) = − ∑…”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “There have been a number of recent improvements on this topic of meta-optimization in multi-task RL (Nichol and Schulman, 2018; Kim et al., 2018; Rothfuss et al., 2018; Flennerhag et al., 2018; Nagabandi et al., 2018; Mendonca et al., 2019; Finn et al., 2019; Lin et al., 2020; Berseth et al., 2021; ”
Meta-Learning with Warped Gradient Descent (2019, background): “WarpGrad is designed for integrating learning paradigms, a promising avenue for future research [28, 27].”
Offline Meta-Reinforcement Learning with Advantage Weighting (2020, methodology): “…into the meta-level optimization (Finn et al., 2017a; Nagabandi et al., 2019; Rothfuss et al., 2018; Zintgraf et al., 2019; Gupta et al., 2018; Mendonca et al., 2019; Yang et al., 2019), potentially using a learned loss function (Houthooft et al., 2018; Bechtle et al., 2019; Kirsch et al.,…”
Meta Learning via Learned Loss (2019, methodology): “In this context, we can divide related work on meta-learning into learning model parameters θ that can be easily adapted to new tasks [7, 18, 10, 35], learning optimizer policies h that transform parameters updates with respect to known loss or reward functions [16, 2, 14, 8, 17, 5], or learning los”
Meta reinforcement learning as task inference (2019, background): “Meta reinforcement learning guided by expert policies has also been studied in a different framework in Mendonca et al. (2019).”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, methodology): “Meta-RL (Duan et al., 2016; Wang et al., 2016; Finn et al., 2017a; Mishra et al., 2017; Rakelly et al., 2019; Mendonca et al., 2019; Zintgraf et al., 2020; Fakoor et al., 2020) and meta-imitation methods (Duan et al.”
Model-based Adversarial Meta-Reinforcement Learning (2020, background): “GMPS [28] improves the sample efficiency during meta-training by consolidating the solutions of individual off-policy learners into a single meta-learner.”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, unknown): “Meta-learning has been both combined with imitation learning [18, 25, 83, 34, 8, 52, 84] and reinforcement learning through context space learning [79, 17, 47, 59, 90] and gradient-based optimization [24, 61, 33, 28, 46].”