Deep Reinforcement Learning amidst Lifelong Non-Stationarity

By Annie Xie, James Harrison, Chelsea Finn. 2020

73 citations recorded by the source corpus.

As humans, our goals and our environment are persistently changing throughout our lifetime based on our experiences, actions, and internal and external drives. In contrast, typical reinforcement learning problem set-ups consider decision processes that are stationary across episodes. Can we develop reinforcement learning algorithms that can cope with the persistent change in the former, more realistic problem settings? While on-policy algorithms such as policy gradients in principle can be extended to non-stationary settings, the same cannot be said for more efficient off-policy algorithms that replay past experiences when learning. In this work, we formalize this problem setting, and draw upon ideas from the online learning and probabilistic inference literature to derive an off-policy RL algorithm that can reason about and tackle such lifelong non-stationarity. Our method leverages latent variable models to learn a representation of the environment from current and past experiences, and performs off-policy RL with this representation. We further introduce several simulation environments that exhibit lifelong non-stationarity, and empirically find that our approach substantially outperforms approaches that do not reason about environment shift.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For example, the recent work of (Xie et al., 2020) considers a formulation with Markovian transitions between tasks which change between episodes but stay constant within an episode.”
Universal Off-Policy Evaluation (2021, methodology): “To do so, we use the following domains: (1) An open source implementation [102] of the FDA-approved type-1 diabetes treatment simulator [59], (2) A stationary and a non-stationary recommender system domain, and (3) A continuous-state Gridworld with partial observability, where data is collected usin”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Xie et al. (2020) is closer to our setting, as they consider nonstationarity within an episode, but they still require manual resets except in a 2D environment which does not contain sink states.”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, unknown): “While some works for lifelong-reinforcement learning [10, 1, 12, 13] or meta-reinforcement learning [3, 67] do aim to address the problem of non-stationarity, they do not provide any safety guarantees.”
Building a Subspace of Policies for Scalable Continual Learning (2022, unknown): “…of these challenges (Tessler et al., 2017; Parisi et al., 2019; Khetarpal et al., 2020; Powers et al., 2021; Wołczyk et al., 2021; Tessler et al., 2017; Berseth et al., 2022; Nagabandi et al., 2018; Xie et al., 2020; Xu et al., 2020; Berseth et al., 2021; Ren et al., 2022; Kessler et al., 2022).”
Sequoia: A Software Framework to Unify Continual Learning Research (2021, methodology): “In RL, this setting is analogous to the DP-MDP [66, 10].”
Block Contextual MDPs for Continual Learning (2021, methodology): “Xie et al. (2020) proposed LILAC that uses a probabilistic hierarchical latent variable model to learn a representation of the environment from current and past experiences and perform off-policy learning.”