Some Considerations on Learning to Explore via Meta-Reinforcement Learning

By Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, Ilya Sutskever. 2018

130 citations recorded by the source corpus.

We consider the problem of exploration in meta reinforcement learning. Two new meta reinforcement learning algorithms are suggested: E-MAML and E-$\text{RL}^2$. Results are presented on a novel environment we call `Krazy World' and a set of maze environments. We show E-MAML and E-$\text{RL}^2$ deliver better performance on tasks where exploration is important.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

CoMPS: Continual Meta Policy Search (2021, methodology): “There are many ways to represent the meta-learned model, including black box models (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2017; Stadie et al., 2018; Rakelly et al., 2019; Zintgraf et al., 2020; Sæmundsson et al., 2018; Fakoor et al., 2019; Sun et al., 2019; Dorfman & Tamar, 2020), ap”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, positive): “Meta-learning has been repeatedly used in the recent literature to learn functions for intrinsic motivation and improved exploration (Baranes and Oudeyer, 2009; Zheng et al., 2018; Stadie et al., 2018; Xu et al., 2018b; Houthooft et al., 2018; Yang et al., 2019; Zou et al., 2019; Zheng et al., 2019)”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “Krazy World [46] 10×10×3 no Randomly generated worlds from the same distribution.”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, neutral): “In contrast to work in multi-task and meta-reinforcement learning [5, 42, 8, 24, 16, 39, 25, 32, 13], our approach learns to use one demonstration to address the meta-exploration problem [11, 36].”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “In contrast to context-based methods, gradient-based metaRL methods learn from aggregated experience using policy gradients (Finn et al., 2017; Stadie et al., 2018; Rothfuss et al., 2018; Xu et al., 2018a), meta-learned loss functions (Sung et al.”
Meta-Learning in Neural Networks: A Survey (2020, background): “Several meta-RL studies have instead explicitly treated exploration strategy or curiosity function as metaknowledge v; and modeled their acquisition as a metalearning problem [26], [183], [184], [228] – leading to sample”
Meta-Reinforcement Learning of Structured Exploration Strategies (2018, background): “Con-currently to our work, Stadie et al. (2017) also explores meta-learning for exploration in an unpublished manuscript, but does not introduce structured stochasticity, and presents results that do not show a signiﬁcant improvement over MAML and other meta-learning algorithms.”
Evolved Policy Gradients (2018, methodology): “Henc , extensions hav be pr posed such as E-RL(2) (37) in which the rewards of episodes sampled early in the learning process are deliberately set to zero to drive exploratory behavior.”
Deep Reinforcement Learning amidst Lifelong Non-Stationarity (2020, methodology): “…meta-reinforcement learning, two dominant techniques exist: optimization-based (Finn et al., 2017; Rothfuss et al., 2019; Zintgraf et al., 2019; Stadie et al., 2018) and contextbased, which includes both recurrent architectures (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2018) and…”
Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy (2022, methodology): “…methods based on MAML (Finn, Abbeel, and Levine 2017) usually learn a pre-trained model from a set of training environments and adapt the pre-trained model to a new environment via several policy gradient steps (Rothfuss et al. 2019; Stadie et al. 2018; Nagabandi et al. 2019; Clavera et al. 2018).”
DiCE: The Infinitely Differentiable Monte-Carlo Estimator (2018, methodology): “Al-Shedivat et al. (2017) and Stadie et al. (2018) derive the proper higher order gradient estimators for their work by reapplying the score function trick.”
Meta Reinforcement Learning with Task Embedding and Shared Policy (2019, background): “Other recent works mainly explore meta-RL from different perspectives about what to meta-learn, such as the exploration ability [Stadie et al. , 2018], the replay buffer for training DDPG [Lillicrap et al. , 2015; Xu et al. , 2018b], non-stationary dynamics [Al-Shedivat et al. , 2017], factors of…”
Concurrent Meta Reinforcement Learning (2019, unknown): “In this paper, we focus on the episodic reinforcement learning meta-learners, particularly those methods outlined in RL2 et al. (2018).”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “These approaches have been extended into the setting of RL (Duan et al., 2016b; Wang et al., 2016; Finn et al., 2017a; Sung et al., 2017; Gupta et al., 2018; Men-donca et al., 2019; Houthooft et al., 2018; Stadie et al., 2018; Rakelly et al., 2019; Nagabandi et al., 2018a).”
ProMP: Proximal Meta-Policy Search (2018, methodology): “…notion of gradient-based Meta-RL, also known as E-MAML, that attempts to circumvent issues with the meta-gradient estimation in MAML (Al-Shedivat et al., 2018; Stadie et al., 2018): Formulation II views U as a deterministic function that depends on N sampled trajectories from a speciﬁc task.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Our agents learn conceptual exploration strategies, something that would require the outer learner of a meta-gradient approach to estimate the return of the inner learner (Stadie et al., 2018).”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, unknown): “Unlike these works and other meta-RL works [61, 13, 38, 46, 18, 30, 26, 47, 56, 58], we close the loop to jointly perform task acquisition and meta-learning so as to achieve an automatic curriculum to facilitate joint meta-learning and task-level exploration.”
Discovery of Useful Questions as Auxiliary Tasks (2019, background): “…et al., 2018), improving few-shot learning performance (Mishra et al., 2018; Duan et al., 2017; Snell et al., 2017), learning to explore (Stadie et al., 2018), unsupervised learning (Gupta et al., 2018; Hsu et al., 2018), few-shot model adaptation (Nagabandi et al., 2018), and improving…”
Deep Reinforcement Learning amidst Continual Structured Non-Stationarity (2021, background): “…meta-reinforcement learning, two dominant techniques exist: optimization-based (Finn et al., 2017; Rothfuss et al., 2019; Zintgraf et al., 2019; Stadie et al., 2018) and context-based, which includes both recurrent architectures (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2018) and…”