Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning
By Christian Alexander Steinparz, Thomas Schmied, Fabian Paischer, Marius-Constantin Dinu, Vihang Patil, Angela Bitto-Nemling, Hamid Eghbalzadeh, Sepp Hochreiter. 2022
19 citations recorded by the source corpus.
In lifelong learning, an agent learns throughout its entire life without resets, in a constantly changing environment, as we humans do. Consequently, lifelong learning comes with a plethora of research problems such as continual domain shifts, which result in non-stationary rewards and environment dynamics. These non-stationarities are difficult to detect and cope with due to their continuous nature. Therefore, exploration strategies and learning methods are required that are capable of tracking the steady domain shifts, and adapting to them. We propose Reactive Exploration to track and react to continual domain shifts in lifelong reinforcement learning, and to update the policy correspondingly. To this end, we conduct experiments in order to investigate different exploration strategies. We empirically show that representatives of the policy-gradient family are better suited for lifelong learning, as they adapt more quickly to distribution shifts than Q-learning. Thereby, policy-gradient methods profit the most from Reactive Exploration and show good results in lifelong learning with continual domain shifts. Our code is available at: https://github.com/ml-jku/reactive-exploration.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matthew Riemer
- Irina Rish
- Khimya Khetarpal
- Doina Precup
- Sergey Levine
- Joelle Pineau
- Eric Eaton
- Chelsea Finn
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “This is generally achieved by defining a heuristic based on some measurement of surprise, information gain, compression, or empowerment (Schmidhuber, 1991; Klyubin et al., 2005; Kaplan and Oudeyer, 2006; Schmidhuber, 2008; Frank et al., 2014; Mohamed and Rezende, 2015; Bellemare et al., 2016; Houtho”