Adapt to Environment Sudden Changes by Learning a Context Sensitive Policy
By Fan Luo, Shengyi Jiang, Yang Yu, Zongzhang Zhang, Yi-Feng Zhang. 2022
45 citations recorded by the source corpus.
Dealing with real-world reinforcement learning (RL) tasks, we shall be aware that the environment may have sudden changes. We expect that a robust policy is able to handle such changes and adapt to the new environment rapidly. Context-based meta reinforcement learning aims at learning environment adaptable policies. These methods adopt a context encoder to perceive the environment on-the-fly, following which a contextual policy makes environment adaptive decisions according to the context. However, previous methods show lagged and unstable context extraction, which are hard to handle sudden changes well. This paper proposes an environment sensitive contextual policy learning (ESCP) approach, in order to improve both the sensitivity and the robustness of context encoding. ESCP is composed of three key components: variance minimization that forces a rapid and stable encoding of the environment context, relational matrix determinant maximization that avoids trivial solutions, and a history-truncated recurrent neural network model that avoids old memory interference. We use a grid-world task and 5 locomotion controlling tasks with changing parameters to empirically assess our algorithm. Experiment results show that in environments with both in-distribution and out-of-distribution parameter changes, ESCP can not only better recover the environment encoding, but also adapt more rapidly to the post-change environment (10x faster in the grid-world) while the return performance is kept or improved, compared with state-of-the-art meta RL methods.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Matthew Riemer
- Irina Rish
- Khimya Khetarpal
- Doina Precup
- Sergey Levine
- Joelle Pineau
- Eric Eaton
- Chelsea Finn
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Changepoint Detection: As discussed previously, one core issue in addressing settings with evolving task states is being able to detect change points or boundaries between significant switches without an oracle as in (Da Silva et al., 2006; Rosman and Ramamoorthy, 2012; Hadoux et al., 2014b; Padakan”