Meta-Reinforcement Learning of Structured Exploration Strategies

By Abhishek Gupta, Russell Mendonca, Pieter Abbeel, Sergey Levine. 2018

391 citations recorded by the source corpus.

Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we explore how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -- model agnostic exploration with structured noise (MAESN) -- to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “In one class of methods (Hausman et al., 2018; Gupta et al., 2018), these variables are held constant over the duration of the episode, providing for a time-correlated exploration signal that can enable a single policy to capture multiple skills and rapidly explore plausible behaviors for new tasks…”
CoMPS: Continual Meta Policy Search (2021, background): “…et al., 2018; Fakoor et al., 2019; Sun et al., 2019; Dorfman & Tamar, 2020), applying gradient descent from initial parameters (Finn et al., 2017; Gupta et al., 2018b; Rothfuss et al., 2018; Clavera et al., 2018; Mendonca et al., 2019; Zintgraf et al., 2019; Mitchell et al., 2020), and training…”
Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, background): “Meta-RL [9, 11, 15, 8] and multi-task RL [27, 36, 40, 43] also seek to accelerate learning by reusing information from different tasks, but in those settings the agent does not handle tasks arriving sequentially and the consequent problem of catastrophic forgetting.”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Much recent met-alearning work [6, 26, 27, 31, 37, 38, 53, 56, 60, 64, 79] focuses on domains where achieving zero-shot generalization is unclear and follow a common paradigm which assumes that the shared representation of monolithic architectures (with the exception of concurrent work [2]) can be…”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “Our results were achieved with ∼ 1 e 6 timesteps while MAESN uses ∼ 1 e 8 timesteps.”
Meta-Learning in Neural Networks: A Survey (2020, background): “However, applications often naturally entail task families which meta-learning can exploit – for example locomotingto or reaching-to different positions [188], navigating within different environments [38], traversing different terrains [65], driving different cars [187], competing with different co”
Meta Learning via Learned Loss (2019, background): “In [8] and [6], a structured latent exploration space is learned from prior experience, which enables fast exploration in novel tasks.”
Meta-learning curiosity algorithms (2020, background): “Gupta et al. (2018) combine gradient-based meta-learning with a learned latent exploration space in which they add structured noise for meaningful exploration.”
Efficient Exploration via State Marginal Matching (2019, background): “Some prior work has explicitly looked at the problem of learning to explore (Gupta et al., 2018; Xu et al., 2018).”
NoRML: No-Reward Meta Learning (2019, neutral): “…learning to update the policy parameters, one critical aspect of the meta-RL problem is the sampling distribution induced by the meta-policy, i.e. how the meta-policy chooses to explore and collect trajectory samples that are maximally informative about the unknown task or environment [12, 27].”
Meta Reinforcement Learning with Task Embedding and Shared Policy (2019, methodology): “However, we observe that simply adapting a single variable is not enough to represent a task in our experiments (conducted in a more challenging way than [Gupta et al. , 2018b]).”
InfoBot: Transfer and Exploration via the Information Bottleneck (2019, background): “Another notable recent exception is Gupta et al. (2018), which took a meta-learning approach to transferable exploration strategies.”
Meta reinforcement learning as task inference (2019, methodology): “Meta learners based on gradients typically utilize the MAML framework [7, 39].”
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning (2019, methodology): “Other prior methods extract reusable skills in the form of latent-variable-conditioned policies, where latent variables can be interpreted as options [38] or abstract skills [20, 18, 12, 17, 13].”
Unsupervised Curricula for Visual Meta-Reinforcement Learning (2019, neutral): “Unlike these works and other meta-RL works [61, 13, 38, 46, 18, 30, 26, 47, 56, 58], we close the loop to jointly perform task acquisition and meta-learning so as to achieve an automatic curriculum to facilitate joint meta-learning and task-level exploration.”
Influence-Based Multi-Agent Exploration (2019, background): “…to boost exploration, including pseudo-counts (Bellemare et al., 2016; Tang et al., 2017; Ostrovski et al., 2017), model-learning improvements (Burda et al., 2018a; Pathak et al., 2017; Burda et al., 2018b), and information gain (Florensa et al., 2017; Gupta et al., 2018; Hyoungseok Kim, 2019).”
Transformers are Meta-Reinforcement Learners (2022, background): “MAESN (Gupta et al., 2018) also creates task variables but optimizes them with on-policy gradient descent and explores by sampling from the prior.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “, 2018), structured exploration strategies (Gupta et al., 2018), transfer feature relevances between families (Kroemer and Sukhatme, 2017),reusable network modules (Alet et al.”