Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting

By Jorge Armando Mendez Mendez, Eric Eaton. 2020

42 citations recorded by the source corpus.

Policy gradient methods have shown success in learning control policies for high-dimensional dynamical systems. Their biggest downside is the amount of exploration they require before yielding high-performing policies. In a lifelong learning setting, in which an agent is faced with multiple consecutive tasks over its lifetime, reusing information from previously seen tasks can substantially accelerate the learning of new tasks. We provide a novel method for lifelong policy gradient learning that trains lifelong function approximators directly via policy gradients, allowing the agent to benefit from accumulated knowledge throughout the entire training process. We show empirically that our algorithm learns faster and converges to better policies than single-task and lifelong learning baselines, and completely avoids catastrophic forgetting on a variety of challenging domains.

Subfields: Artificial Intelligence, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “A distinctive feature of CRL is its high diversity of applications and benchmarks, including continuous control [203], [279], [296], [449], maze navigation [274], [370], video games (Atari [10], [59], [195], [220], [370], [429], StarCraft [372], Minecraft [405], Hanabi [312]), etc.”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, methodology): “Our method also outperforms LPG-FTW [43] which uses 2.”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “[33] use Meta-World tasks, similarly to us, for evaluations of their continual learning method, but the work is not aimed at building a benchmark.”
Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning (2022, background): “In continual RL, a task z constitutes a stationary Markov decision process (MDP) M (z) = 〈S(z), A(z), p(z), r(z), γ(z)〉 (Khetarpal et al., 2020; Mendez et al., 2020).”
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 (2022, background): “, (Mendez et al., 2020)) have been tested on benchmark problems, but proved to be challenging to scale up to the complex observation and action space of SC-2.”
Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, background): “For example, Ammar et al. (2014) and Mendez et al. (2020) assign task-speciﬁc parameters to each task while sharing a reusable knowledge base among all tasks.”
Sequoia: A Software Framework to Unify Continual Learning Research (2021, methodology): “In Table 2, we apply the continual-world methods, built on top of SAC, on a incremental RL benchmark inspired by Mendez et al. (2020) (see App.”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “Mendez et al. (2020) developed LPG-FTW, a mechanism that uses multiple models like PG-ELLA variants, but learns these models directly via RL training like single-model methods.”