Evolved Policy Gradients
By Rein Houthooft, Richard Y. Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, Pieter Abbeel. 2018
238 citations recorded by the source corpus.
We propose a metalearning approach for learning gradient-based reinforcement learning (RL) algorithms. The idea is to evolve a differentiable loss function, such that an agent, which optimizes its policy to minimize this loss, will achieve high rewards. The loss is parametrized via temporal convolutions over the agent's experience. Because this loss is highly flexible in its ability to take into account the agent's history, it enables fast task learning. Empirical results show that our evolved policy gradient algorithm (EPG) achieves faster learning on several randomized environments compared to an off-the-shelf policy gradient method. We also demonstrate that EPG's learned loss can generalize to out-of-distribution test time tasks, and exhibits qualitatively different behavior from other popular metalearning algorithms.
Subfields: Artificial Intelligence, Reinforcement Learning, Evolutionary Computation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Chelsea Finn
- Pieter Abbeel
- Quoc V. Le
- Ruslan Salakhutdinov
- Mohit Bansal
- Dacheng Tao
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Abhishek Gupta
- Ofir Nachum
- Yuxi Li
- Shixiang Shane Gu
In Evolutionary Computation
- Shimon Whiteson
- Sergey Levine
- Chelsea Finn
- Jian Peng
- Chen Liang
- J. Urgen Schmidhuber
- Satinder Singh
- Yoshua Bengio
Citations in the corpus
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “, 2019) or the objective function in reinforcement learning (Houthooft et al., 2018; Kirsch et al., 2019; Alet et al., 2019); in these works, all other algorithmic aspects remain hand-designed.”
- CoMPS: Continual Meta Policy Search (2021, background): “…(Finn et al., 2017; Gupta et al., 2018b; Rothfuss et al., 2018; Clavera et al., 2018; Mendonca et al., 2019; Zintgraf et al., 2019; Mitchell et al., 2020), and training a critic to provide policy gradients (Sung et al., 2017; Houthooft et al., 2018; Yu et al., 2018; Chebotar et al., 2019).”
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, positive): “Meta-learning has been repeatedly used in the recent literature to learn functions for intrinsic motivation and improved exploration (Baranes and Oudeyer, 2009; Zheng et al., 2018; Stadie et al., 2018; Xu et al., 2018b; Houthooft et al., 2018; Yang et al., 2019; Zou et al., 2019; Zheng et al., 2019)”
- Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “additionally improve upon trial-and-error experience. In contrast to work in multi-task and meta-reinforcement learning (Duan et al., 2016; Wang et al., 2016; Finn et al., 2017a; Mishra et al., 2018; Houthooft et al., 2018; Sung et al., 2017; Nagabandi et al., 2019; Sæmundsson et al., 2018; Hausman ”
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “Most current meta-RL methods require on-policy data during both meta-training and adaptation (Finn et al., 2017; Wang et al., 2016; Duan et al., 2016; Mishra et al., 2018; Rothfuss et al., 2018; Houthooft et al., 2018), which makes them exceedingly inefficient during meta-training.”
- Meta-Learning in Neural Networks: A Survey (2020, methodology): “Successful applications have been demonstrated in areas spanning few-shot image recognition [19], [23], unsupervised learning [16], data efficient [24], [25] and self-directed [26] reinforcement learning (RL), hyperparameter optimization [20], and neural architecture search (NAS) [21], [27], [28].”
- Editable Neural Networks (2020, methodology): “These methods were shown to be extremely successful in a large number of problems, such as few-shot learning (Finn et al. (2017); Nichol et al. (2018)), learnable optimization (Andrychowicz et al. (2016)) and reinforcement learning (Houthooft et al. (2018)).”
- Online Meta-Critic Learning for Off-Policy Actor-Critic Methods (2020, methodology): “, 2019), loss functions (Houthooft et al., 2018; Sung et al., 2017) or rewards (Zheng et al.”
- Offline Meta-Reinforcement Learning with Advantage Weighting (2020, background): “, 2019), potentially using a learned loss function (Houthooft et al., 2018; Bechtle et al., 2019; Kirsch et al., 2020).”
- NoRML: No-Reward Meta Learning (2019, unknown): “Prior model-free meta reinforcement learning algorithms can generally be categorized as being recurrence-based [8, 19, 27, 33], gradientbased [10, 12, 27], or a hybrid of the two [15, 28].”
- Introducing Symmetries to Black Box Meta Reinforcement Learning (2021, background): “2020), meta-learning hyperparameters (Sutton 1992; Xu, van Hasselt, and Silver 2018) or meta-learning objective functions corresponding to the learning algorithm (Houthooft et al. 2018; Kirsch, van Steenkiste, and Schmidhuber 2020; Xu et al. 2020; Oh et al. 2020; Bechtle et al. 2021).”
- Self-Attentional Credit Assignment for Transfer in Reinforcement Learning (2019, unknown): “Others learn auxiliary reward functions in the hope that they will enable transfer by imposing consistency in the reward [Houthooft et al. , 2019].”
- Fast and Slow Learning of Recurrent Independent Mechanisms (2021, background): “…the goal of achieving fast adaptation to changes in distribution or a new task in a reinforcement-learning agent (Duan et al., 2016; Mishra et al., 2017; Wang et al., 2016; Nichol et al., 2018; Xu et al., 2018; Houthooft et al., 2018; Kirsch et al., 2019; Botvinick et al., 2019; Oh et al., 2020).”
- Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, background): “…for fast adaptation to new environments is meta reinforcement learning (meta RL) (Cully et al., 2015; Finn et al., 2017; Wang et al., 2017; Duan et al., 2016; Xu et al., 2018; Houthooft et al., 2018; Sæmundsson et al., 2018; Nagabandi et al., 2018; Humplik et al., 2019; Rakelly et al., 2019).”
- Edge-Labeling Graph Neural Network for Few-Shot Learning (2019, background): “A lot of interest in meta-learning [1] has been recently arisen in various areas including especially taskgeneralization problems such as few-shot learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], learn-to-learn [16, 17, 18], non-stationary reinforcement learning[19, 20, 21], and continual ”
- Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes (2020, methodology): “Meta-learning algorithms in general consist of a base (quick) learner and a meta (slow) learner [18, 19] and have recently been combined with RL to solve nonstationary problems via model-free [20, 21, 22] and model-based approaches [3, 2].”
- Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning (2018, methodology): “ons using meta-learning. Our approach is similar as we also consider the problem of learning to adapt. However, we consider the problem of visual navigation and adapt via a self-supervised loss. Both [18] and [48] learn an objective function. However, [18] use evolutionary strategies instead of meta”
- Guided Meta-Policy Search (2019, unknown): “However, despite recent advances (Duan et al., 2016; Finn et al., 2017a; Houthooft et al., 2018), current meta-RL methods are generally limited to much simpler domains, such as relatively low-dimensional continuous control tasks (Finn et al., 2017a; Sung et al., 2017) and navigation with discrete…”
- Deep Curiosity Search: Intra-Life Exploration Improves Performance on Challenging Deep Reinforcement Learning Problems (2018, background): “Many algorithms have tried to produce better exploration in RL [7, 9, 10, 11, 14, 15, 16, 17, 18].”