Adaptive Online Planning for Continual Lifelong Learning

By Kevin Lu, Igor Mordatch, Pieter Abbeel. 2019

15 citations recorded by the source corpus.

We study learning control in an online reset-free lifelong learning scenario, where mistakes can compound catastrophically into the future and the underlying dynamics of the environment may change. Traditional model-free policy learning methods have achieved successes in difficult tasks due to their broad flexibility, but struggle in this setting, as they can activate failure modes early in their lifetimes which are difficult to recover from and face performance degradation as dynamics change. On the other hand, model-based planning methods learn and adapt quickly, but require prohibitive levels of computational resources. We present a new algorithm, Adaptive Online Planning (AOP), that achieves strong performance in this setting by combining model-based planning with model-free learning. By approximating the uncertainty of the model-free components and the planner performance, AOP is able to call upon more extensive planning only when necessary, leading to reduced computation times, while still gracefully adapting behaviors in the face of unpredictable changes in the world -- even when traditional RL fails.

Subfields: Artificial Intelligence, Management Science and Operations Research, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “A distinctive feature of CRL is its high diversity of applications and benchmarks, including continuous control [203], [279], [296], [449], maze navigation [274], [370], video games (Atari [10], [59], [195], [220], [370], [429], StarCraft [372], Minecraft [405], Hanabi [312]), etc.”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “Continuous control [31, 23] use continuous control tasks such as Humanoid or Walker2D.”
Optimizing for the Future in Non-Stationary MDPs (2020, background): “Other works have also shown how models of the environment can be used for continual learning (Lu et al., 2019) or using it along with a model predictive control (Wagener et al.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Lu et al. (2019) and Lowrey et al. (2018) highlight the beneﬁts of both planning and model-free RL for lifelong agents but require highly accurate world models.”