Learning Domain Invariant Representations in Goal-conditioned Block MDPs
By Beining Han, Chongyi Zheng, Harris Chan, Keiran Paster, Michael R. Zhang, Jimmy Ba. 2021
19 citations recorded by the source corpus.
Deep Reinforcement Learning (RL) is successful in solving many complex Markov Decision Processes (MDPs) problems. However, agents often face unanticipated environmental changes after deployment in the real world. These changes are often spurious and unrelated to the underlying problem, such as background shifts for visual input agents. Unfortunately, deep RL policies are usually sensitive to these changes and fail to act robustly against them. This resembles the problem of domain generalization in supervised learning. In this work, we study this problem for goal-conditioned RL agents. We propose a theoretical framework in the Block MDP setting that characterizes the generalizability of goal-conditioned policies to new environments. Under this framework, we develop a practical method PA-SkewFit that enhances domain generalization. The empirical evaluation shows that our goal-conditioned RL agent can perform well in various unseen test environments, improving by 50% over baselines.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Amy Zhang
- Matthew Riemer
- Doina Precup
- Chelsea Finn
- Irina Rish
- Ashvin Nair
- Pieter Abbeel
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Additionally, Block MDPs have been formalized in settings with multiple domains (Han et al., 2021).”
- Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, background): “Han et al. (2021) address the generalization problem of goal-conditioned policies to novel observations at test time by using an alignment objective for data collection across environments.”