Reset-Free Lifelong Learning with Skill-Space Planning
By Kevin Lu, Aditya Grover, Pieter Abbeel, Igor Mordatch. 2020
44 citations recorded by the source corpus.
The objective of lifelong reinforcement learning (RL) is to optimize agents which can continuously adapt and interact in changing environments. However, current RL approaches fail drastically when environments are non-stationary and interactions are non-episodic. We propose Lifelong Skill Planning (LiSP), an algorithmic framework for non-episodic lifelong RL based on planning in an abstract space of higher-order skills. We learn the skills in an unsupervised manner using intrinsic rewards and plan over the learned skills using a learned dynamics model. Moreover, our framework permits skill discovery even from offline data, thereby reducing the need for excessive real-world interactions. We demonstrate empirically that LiSP successfully enables long-horizon planning and learns agents that can avoid catastrophic failures even in challenging non-stationary and non-episodic environments derived from gridworld and MuJoCo benchmarks.
Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhishek Gupta
- Aravind Rajeswaran
- Igor Mordatch
- Ruslan Salakhutdinov
- Yoshua Bengio
In Management Science and Operations Research
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Aravind Rajeswaran
- Abhishek Gupta
- Igor Mordatch
- Kevin Lu
- Doina Precup
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhishek Gupta
- Aravind Rajeswaran
- Igor Mordatch
- Vikash Kumar
- Shixiang Shane Gu
Citations in the corpus
- Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “, 2021) or by maximizing mutual information (Eysenbach et al., 2019; Lu et al., 2020; Sharma et al., 2020).”
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Meanwhile, (Lu et al., 2020) have proposed a skill space planning framework for continual RL environments with no resets.”
- How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “A recent model-based approach learns skills in an off-line phase that subsequently enable the agent to learn in a nonstationary lifetime without explicit tasks (Lu et al., 2021).”
- Autonomous Reinforcement Learning via Subgoal Curricula (2021, background): “Other recent work like [29] has considered combining model-based RL with unsupervised skill discovery to solve reset-free learning problems, but largely focus on avoiding sink states rather than attempting tasks repeatedly with a curriculum like VaPRL.”
- Building a Subspace of Policies for Scalable Continual Learning (2022, background): “There is also a large body of work which leverages the shared structure of the tasks [50, 30, 25, 27, 1, 43], meta-learning [19, 44, 4, 8, 33, 52, 39], or generative models [35, 42, 41, 2] to improve CRL agents, but these are less related to CSP and not very competitive.”