Continual Reinforcement Learning in 3D Non-stationary Environments

By Vincenzo Lomonaco, Karan Desai, Eugenio Culurciello, Davide Maltoni. 2019

49 citations recorded by the source corpus.

High-dimensional always-changing environments constitute a hard challenge for current reinforcement learning techniques. Artificial agents, nowadays, are often trained off-line in very static and controlled conditions in simulation such that training observations can be thought as sampled i.i.d. from the entire observations space. However, in real world settings, the environment is often non-stationary and subject to unpredictable, frequent changes. In this paper we propose and openly release CRLMaze, a new benchmark for learning continually through reinforcement in a complex 3D non-stationary task based on ViZDoom and subject to several environmental changes. Then, we introduce an end-to-end model-free continual reinforcement learning strategy showing competitive results with respect to four different baselines and not requiring any access to additional supervised signals, previously encountered environmental conditions or observations.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “A distinctive feature of CRL is its high diversity of applications and benchmarks, including continuous control [203], [279], [296], [449], maze navigation [274], [370], video games (Atari [10], [59], [195], [220], [370], [429], StarCraft [372], Minecraft [405], Hanabi [312]), etc.”
DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “In the context of continual reinforcement learning, several approaches have been proposed, such as the use of Progressive Nets in [Rusu et al., 2016b], EWC [Kirkpatrick et al., 2017], Progress And Compress (P&C) [Schwarz et al., 2018], or CRL-Unsup [Lomonaco et al., 2019].”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “[29] propose CRLMaze, 3D navigation scenarios for continual learning, which solely concentrate on changes of the visual aspects.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “Lomonaco et al. (2020) designed CRLMaze based on VizDoom (Kempka et al., 2016), an object-picking LLL task that is composed of 4 scenarios (Light, Texture, Object, All) of incremental difﬁculty and a total of 12 maps.”
Continual Learning for Robotics (2019, background): “, 2018), CRLMaze extension of VizDoom (Lomonaco et al., 2019) and DeepMind Lab (Mankowitz et al.”
Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning (2020, methodology): “…on locally stationary data distributions, e.g., an autonomous vehicle navigating speciﬁc weather, a recommendation system dealing with users with time-varying interest (Hidasi et al., 2015), or an embedded agent doing a speciﬁc task for a while (Lomonaco et al., 2019a; Padakandla et al., 2019).”
Avalanche: an End-to-End Library for Continual Learning (2021, background): “The Evaluator object directly interacting with the algorithm provides a uniﬁed interface to control and compute several performance metrics ( p i ), delegating results logging to the Logger(s) objects. to Robotics [28, 40, 38], from Reinforcement Learning [25, 31] to Sequence Learning [8], among oth”
Online Continual Learning on Sequences (2020, methodology): “• Reinforcement Learning : Arcade Learning Environment (ALE) [3] for Atari games, SURREAL [11] for robot manipulation and RoboTurk for robotic skill learning through imitation [49], CRLMaze extension of VizDoom [39] and DeepMind Lab [50].”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, background): “Other video game-like environments proposed to evaluate continual learning include StarCraft (Schwarz et al., 2018a) and VizDoom (Lomonaco et al., 2020).”
Continual Learning in Neural Networks (2019, background): “Only very recently new research works have started to explore this setting [54, 91].”