Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret

By Haitham Bou-Ammar, Rasul Tutunov, Eric Eaton. 2015

68 citations recorded by the source corpus.

Lifelong reinforcement learning provides a promising framework for developing versatile agents that can accumulate knowledge over a lifetime of experience and rapidly learn new tasks by building upon prior knowledge. However, current lifelong learning methods exhibit non-vanishing regret as the amount of experience increases, and include limitations that can lead to suboptimal or unsafe control policies. To address these issues, we develop a lifelong policy gradient learner that operates in an adversarial setting to learn multiple tasks online while enforcing safety constraints on the learned policies. We demonstrate, for the first time, sublinear regret for lifelong policy search, and validate our algorithm on several benchmark dynamical systems and an application to quadrotor control.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Constrained Policy Optimization (2017, methodology): “Bou Ammar et al. (2015) propose a theoretically-motivated policy gradient method for lifelong learning with safety constraints, but their method involves an expensive inner loop optimization of a semi-deﬁnite program, making it unsuited for the deep RL setting.”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “For example, Ammar et al. (2014) and Ammar et al. (2015) transfer knowledge via a latent basis whereas Brunskill and Li (2014) perform batch optimization across all encountered tasks.”
Hierarchical reinforcement learning for efficient exploration and transfer (2020, background): “In case prior knowledge of S i × A i is not available, previous work on lifelong learning has shown how to automatically learn a latent state space that is common to a range of tasks (Bou Ammar et al., 2015).”
Adaptive Skills Adaptive Partitions (ASAP) (2016, background): “These are the necessary requirements for a truly general skill learning framework and can be applied to lifelong learning problems (Ammar et al., 2015; Thrun & Mitchell, 1995).”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, unknown): “[4], which aims to find a policy that satisfies the safety constraints in the lifelong-learning setting.”
Unicorn: Continual Learning with a Universal, Off-policy Agent (2018, background): “In a linear RL setting Ammar et al. [1] , learn a latent basis via policy gradients to solve new tasks as they are encountered.”