Continual State Representation Learning for Reinforcement Learning using Generative Replay

By Hugo Caselles-Dupré, Michaël Garcia Ortiz, David Filliat. 2018

20 citations recorded by the source corpus.

We consider the problem of building a state representation model in a continual fashion. As the environment changes, the aim is to efficiently compress the sensory state's information without losing past knowledge. The learned features are then fed to a Reinforcement Learning algorithm to learn a policy. We propose to use Variational Auto-Encoders for state representation, and Generative Replay, i.e. the use of generated samples, to maintain past knowledge. We also provide a general and statistically sound method for automatic environment change detection. Our method provides efficient state representation as well as forward transfer, and avoids catastrophic forgetting. The resulting model is capable of incrementally learning information without using past data and with a bounded system size.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

Citations in the corpus

GAN Memory with No Forgetting (2020, background): “However, most existing generative replay methods either deliver blurry generated samples [10, 40] or only work well on simple datasets [40, 78, 40] like MNIST; besides, they often don’t scale well to practical situations with high resolution [60] or a long sequence [86], sometimes even with negative”
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning (2019, background): “[1] replay the seen categories with GANs [6].”
Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting (2018, methodology): “To our knowledge, pseudo-rehearsal has only been applied by [38] to sequentially learn reinforcement tasks.”
Dynamics-aware Embeddings (2019, background): “Several papers have proposed using (variational) auto-encoders to learn embeddings for observations (Lange & Riedmiller, 2010; Van Hoof et al., 2016; Higgins et al., 2017b; Caselles-Dupré et al., 2018); unlike our work, these models operate on a single observation at a time and do not depend on the ”