Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation

By Shani Gamrian, Yoav Goldberg. 2018

127 citations recorded by the source corpus.

Despite the remarkable success of Deep RL in learning control policies from raw pixels, the resulting models do not generalize. We demonstrate that a trained agent fails completely when facing small visual changes, and that fine-tuning---the common transfer learning paradigm---fails to adapt to these changes, to the extent that it is faster to re-train the model from scratch. We show that by separating the visual transfer task from the control policy we achieve substantially better sample efficiency and transfer behavior, allowing an agent trained on the source task to transfer well to the target tasks. The visual mapping from the target to the source domain is performed using unaligned GANs, resulting in a control policy that can be further improved using imitation learning from imperfect demonstrations. We demonstrate the approach on synthetic visual variants of the Breakout game, as well as on transfer between subsequent levels of Road Fighter, a Nintendo car-driving game. A visualization of our approach can be seen in this https URL and this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “…in gameplay (Mnih et al., 2015; Silver et al., 2017; Berner et al., 2019), it is found that agents are fragile under distributional shift (Kansky et al., 2017; Zhang et al., 2018; Gamrian and Goldberg, 2019) and require substantially more training data than humans (Tsividis et al., 2017).”
Domain Generalization with MixStyle (2021, background): “Gamrian & Goldberg (2019) leveraged the advances in GAN-based image-to-image translation (Liu et al., 2017) to map target data to the source domain which the agent was trained on.”
AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, methodology): “Moreover, in the scenario of partially observable Markov decision processes (POMDP), most approaches do transfer on the image level, but fail to understand the changes of the underlying states, making transfer hard and unreliable [Gamrian and Goldberg, 2018]. Considering these limitations, we propos”
Image-to-Image Translation: Methods and Applications (2021, methodology): “transfer learning \ \ [217] \ \ \ \ \ \ \ \ image registration \ \ [38] \ \ \ \ \ \ \ \ domain adaptation \ \ [29], [30] \ \ \ [31] \ \ \ \”
Observational Overfitting in Reinforcement Learning (2019, background): “Several artiﬁcial benchmarks (Zhang et al., 2018b; Gamrian & Goldberg, 2019) have been proposed before to portray this notion of overﬁtting, where an agent must deal with a changing background - however, a key difference in our work is that we explicitly require the “background” to be correlated…”
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning (2019, background): “However, it has been evidenced in recent years that deep RL agents often struggle to generalize to new environments, even when semantically similar to trained agents (Farebrother et al., 2018; Zhang et al., 2018b; Gamrian & Goldberg, 2019; Cobbe et al., 2019).”
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning (2020, background): “A series of recent studies show that RL agents fail to generalize to new environments, even when they are trained on a large set of diverse yet semantically similar ones [17, 19, 59, 11, 12, 50].”