ProMP: Proximal Meta-Policy Search

By Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, Pieter Abbeel. 2018

227 citations recorded by the source corpus.

Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-efficiency during meta-training as well as ineffective task identification strategies. This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL. Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difficulties in estimating meta-policy gradients. By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efficient and stable meta-learning. Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CoMPS: Continual Meta Policy Search (2021, methodology): “, 2019; Dorfman & Tamar, 2020), applying gradient descent from initial parameters (Finn et al., 2017; Gupta et al., 2018b; Rothfuss et al., 2018; Clavera et al., 2018; Mendonca et al., 2019; Zintgraf et al., 2019; Mitchell et al., 2020), and training a critic to provide policy gradients (Sung et al.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “There have been a number of recent improvements on this topic of meta-optimization in multi-task RL (Nichol and Schulman, 2018; Kim et al., 2018; Rothfuss et al., 2018; Flennerhag et al., 2018; Nagabandi et al., 2018; Mendonca et al., 2019; Finn et al., 2019; Lin et al., 2020; Berseth et al., 2021; ”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, methodology): “From Figure 5 and Table 1, we find that the prior meta-RL methods, MAML and RL2 reach 36% and 10% success on ML10 test tasks, while PEARL is unable to generalize to new tasks on ML10.”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “…methods, gradient-based meta-RL methods learn from aggregated experience using policy gradients (Finn et al., 2017; Stadie et al., 2018; Rothfuss et al., 2018; Xu et al., 2018a), meta-learned loss functions (Sung et al., 2017; Houthooft et al., 2018), or hyperparameters (Xu et al.,…”
Meta-Learning in Neural Networks: A Survey (2020, background): “meta-optimization algorithms [167], [168], [169] for these tasks, and provided theoretical guarantees formeta-RL [225].”
Meta-Curvature (2019, background): “, 2018), better credit assignment (Rothfuss et al., 2019), and robustness (Kim et al.”
Model-based Adversarial Meta-Reinforcement Learning (2020, background): “ProMP [39] improves the sample-efficiency during meta-training by overcoming the issue of poor credit assignment.”
Toward Robust Long Range Policy Transfer (2021, background): “First, unlike supervised learning, the transitions and trajectories are sampled during the training phase based on the interacted policy (Rothfuss et al. 2019).”
Guided Meta-Policy Search (2019, methodology): “…al., 2016; Mishra et al., 2018; Stadie et al., 2018), using gradient descent from a learned initialization (Finn et al., 2017a; Gupta et al., 2018; Rothfuss et al., 2018), using a learned critic that provides gradients to the policy (Sung et al., 2017; Houthooft et al., 2018), or using a planner…”
Multi-Step Model-Agnostic Meta-Learning: Convergence and Improved Algorithms (2020, background): “Many follow-up studies (Grant et al., 2018; Finn et al., 2019; Jerfel et al., 2018; Finn and Levine, 2017; Finn et al., 2018; Mi et al., 2019; Liu et al., 2019; Rothfuss et al., 2018; Foerster et al., 2018; Fallah et al., 2019; Collins et al., 2020; Fallah et al., 2020) have extended MAML from diffe”
Skill-based Meta-Reinforcement Learning (2022, background): “Meta-RL approaches (Duan et al., 2016; Wang et al., 2017; Finn et al., 2017; Yu et al., 2018; Rothfuss et al., 2019; Gupta et al., 2018; Vuorio et al., 2018; Nagabandi et al., 2019; Clavera et al., 2019; 2018; Rakelly et al., 2019; Vuorio et al., 2019; Yang et al., 2019; Zintgraf et al., 2019;…”
VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2019, methodology): “The other methods (PEARL Rakelly et al. (2019), E-MAML Stadie et al. (2018) and ProMP Rothfuss et al. (2019)) are not designed to maximise reward during a single rollout, and perform poorly in this case.”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, unknown): “Unlike these works and other meta-RL works [61, 13, 38, 46, 18, 30, 26, 47, 56, 58], we close the loop to jointly perform task acquisition and meta-learning so as to achieve an automatic curriculum to facilitate joint meta-learning and task-level exploration.”
Deep Reinforcement Learning amidst Continual Structured Non-Stationarity (2021, methodology): “Within meta-reinforcement learning, two dominant techniques exist: optimization-based (Finn et al., 2017; Rothfuss et al., 2019; Zintgraf et al., 2019; Stadie et al., 2018) and context-based, which includes both recurrent architectures (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2018) and…”
Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning (2020, methodology): “Some previous approaches (Rakelly et al., 2019; Rothfuss et al., 2019) utilize Thompson-sampling (Schmidhuber, 1987) to explore, in which an agent needs to explore in the initial few episodes and execute optimally in the subsequent episode.”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “In the more general case with arbitrary reward functions and arbitrary learning procedures, intermediate rewards along a trajectory may be informative, and the optimal exploration strategy may be different from posterior sampling (Rothfuss et al., 2019; Duan et al., 2016b; Wang et al., 2016).”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, unknown): “Meta-learning has been both combined with imitation learning [18, 25, 83, 34, 8, 52, 84] and reinforcement learning through context space learning [79, 17, 47, 59, 90] and gradient-based optimization [24, 61, 33, 28, 46].”
Transformers are Meta-Reinforcement Learners (2022, background): “These locomotion tasks are previously introduced by Finn et al. (2017) and Rothfuss et al. (2018).”