Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes

By Mengdi Xu, Wenhao Ding, Jiacheng Zhu, Zuxin Liu, Baiming Chen, Ding Zhao. 2020

38 citations recorded by the source corpus.

Continuously learning to solve unseen tasks with limited experience has been extensively pursued in meta-learning and continual learning, but with restricted assumptions such as accessible task distributions, independently and identically distributed tasks, and clear task delineations. However, real-world physical tasks frequently violate these assumptions, resulting in performance degradation. This paper proposes a continual online model-based reinforcement learning approach that does not require pre-training to solve task-agnostic problems with unknown task boundaries. We maintain a mixture of experts to handle nonstationarity, and represent each different type of dynamics with a Gaussian Process to efficiently leverage collected data and expressively model uncertainty. We propose a transition prior to account for the temporal dependencies in streaming data and update the mixture online via sequential variational inference. Our approach reliably handles the task distribution shift by generating new models for never-before-seen dynamics and reusing old models for previously seen dynamics. In experiments, our approach outperforms alternative methods in non-stationary tasks, including classic control with changing dynamics and decision making in different driving scenarios.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “…synthetic non-stationarity to the environments (Henderson et al., 2017; Al-Shedivat et al., 2017) or ordering some completely unrelated environments in a sequence (Xu et al., 2020) to facilitate the evaluation of LLL performance (eg. a random sequence of Atari used in (Kirk-patrick et al., 2017)).”
Building a Subspace of Policies for Scalable Continual Learning (2022, unknown): “…of these challenges (Tessler et al., 2017; Parisi et al., 2019; Khetarpal et al., 2020; Powers et al., 2021; Wołczyk et al., 2021; Tessler et al., 2017; Berseth et al., 2022; Nagabandi et al., 2018; Xie et al., 2020; Xu et al., 2020; Berseth et al., 2021; Ren et al., 2022; Kessler et al., 2022).”
Continual Model-Based Reinforcement Learning with Hypernetworks (2020, unknown): “The learning curve is omitted here due to space constraint and is available on our project website.”