Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning

By Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, Chelsea Finn. 2018

659 citations recorded by the source corpus.

Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.

Subfields: Artificial Intelligence, Biomedical Engineering, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks (2020, background): “In other scenarios, models must adapt efficiently to changing environments before enough time has passed to amass a large and diverse data corpus (Nagabandi et al., 2018).”
Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (2020, background): “Meta-RL [9, 11, 15, 8] and multi-task RL [27, 36, 40, 43] also seek to accelerate learning by reusing information from different tasks, but in those settings the agent does not handle tasks arriving sequentially and the consequent problem of catastrophic forgetting.”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “This is why recent reinforcement learning algorithms for continual learning have been tested only on arguably simple tasks of low/medium input space dimension and complexity [21, 1, 35].”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, methodology): “n [18,19,20], continuous control domains with parametric variation across tasks [10,21,22,23], bandit problems [19,18,20,24], levels of an arcade game [25], and locomotion tasks with varying dynamics [26,27]. Complementary to these evaluations, we aim to develop a testbed of tasks and an evaluation ”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “Note that the planning procedure is based on stochastic optimization, following prior work (Nagabandi et al., 2018), and we provide more details in the appendix.”
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (2019, methodology): “Recently, meta-RL methods have been developed for meta-learning dynamics models (Nagabandi et al., 2019; Sæ-mundsson et al., 2018) and policies (Finn et al., 2017; Duan et al., 2016; Mishra et al., 2018) that can quickly adapt to new tasks.”
Continuous Meta-Learning without Tasks (2019, methodology): “These baselines are a competitive approach to learning in time-varying data streams [13] and have been applied meta-learning in time-varying settings [31].”
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, methodology): “Similar to previous methods (Nagabandi et al., 2018a; Rakelly et al., 2019; Zhou et al., 2019; Lee et al., 2020; Seo et al., 2020), we use the past transition segments After obtaining environment-speciﬁed ˆ z t − k : t − 1 at timestep t , we incorporate it into the dynamics prediction model ˆ f to…”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Adaptation also plays a critical role in robotics, with agents trained to adapt to varying terrain (Clavera et al., 2019; Kumar et al., 2021) or damaged joints (Cully et al.”
A CCELERATING O NLINE R EINFORCEMENT L EARNING VIA M ODEL -B ASED M ETA -L EARNING (2021, methodology): “MAML for model-based RL has been successful on low-dimensional tasks (Nagabandi et al., 2019b;a) with limited horizons but has not scaled to more complex image based domains where : Here we show that as an agent interacts with the world over a lifetime (blue line) it gains diverse experience across…”
Continual Backprop: Stochastic Gradient Descent with Persistent Randomness (2021, background): “Multiple recent works have focused on adapting to the changes in the data stream by ﬁnding new useful information (Rusu et al. 2016, Finn et al. 2017, Wang et al. 2017; Nagabandi et al. 2019, Finn et al. 2019).”