A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning
By Jixiang Guo, Mingming Gong, Dacheng Tao. 2022
21 citations recorded by the source corpus.
The generalization of model-based reinforcement learning (MBRL) methods to environments with unseen transition dynamics is an important yet challenging problem. Existing methods try to extract environment-specified information $Z$ from past transition segments to make the dynamics prediction model generalizable to different dynamics. However, because environments are not labelled, the extracted information inevitably contains redundant information unrelated to the dynamics in transition segments and thus fails to maintain a crucial property of $Z$: $Z$ should be similar in the same environment and dissimilar in different ones. As a result, the learned dynamics prediction function will deviate from the true one, which undermines the generalization ability. To tackle this problem, we introduce an interventional prediction module to estimate the probability of two estimated $\hat{z}_i, \hat{z}_j$ belonging to the same environment. Furthermore, by utilizing the $Z$'s invariance within a single environment, a relational head is proposed to enforce the similarity between $\hat{{Z}}$ from the same environment. As a result, the redundant information will be reduced in $\hat{Z}$. We empirically show that $\hat{{Z}}$ estimated by our method enjoy less redundant information than previous methods, and such $\hat{{Z}}$ can significantly reduce dynamics prediction errors and improve the performance of model-based RL methods on zero-shot new environments with unseen dynamics. The codes of this method are available at \url{https://github.com/CR-Gjx/RIA}.
Subfields: Artificial Intelligence, Biomedical Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Biomedical Engineering
Citations in the corpus
- Continual Learning through Retrieval and Imagination (2022, methodology): “In this section, we analyze the generalization of our learning framework based on robustness theory in machine learning (Bellet et al. 2015; Mohri et al. 2009; Yin et al. 2020; Guo et al. 2022).”
- Continual Learning with Lifelong Vision Transformer (2022, background): “Structure-based meth-ods [1, 29, 43, 49, 50, 57, 68, 87] expand networks as new tasks arrive and keep the parameters of sub-networks re-lated to previous tasks fixed.”