Unsupervised Cross-Domain Transfer in Policy Gradient Reinforcement Learning via Manifold Alignment
By Haitham Bou-Ammar, Eric Eaton, Paul Ruvolo, Matthew E. Taylor. 2015
96 citations recorded by the source corpus.
The success of applying policy gradient reinforcement learning (RL) to difficult control tasks hinges crucially on the ability to determine a sensible initialization for the policy. Transfer learning methods tackle this problem by reusing knowledge gleaned from solving other related tasks. In the case of multiple task domains, these algorithms require an inter-task mapping to facilitate knowledge transfer across domains. However, there are currently no general methods to learn an inter-task mapping without requiring either background knowledge that is not typically present in RL settings, or an expensive analysis of an exponential number of inter-task mappings in the size of the state and action spaces. This paper introduces an autonomous framework that uses unsupervised manifold alignment to learn inter-task mappings and effectively transfer samples between different task domains. Empirical results on diverse dynamical systems, including an application to quadrotor control, demonstrate its effectiveness for cross-domain transfer in the context of policy gradient RL.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Learning to Generalize: Meta-Learning for Domain Generalization (2017, neutral): “2017)), or un-rewarded (≈ unsupervised domain adaptation (Finn et al. 2017; Ammar et al. 2015)) trials in the target environment for adaptation.”
- PERCEPTUAL GOAL SPECIFICATIONS FOR REINFORCEMENT LEARNING (2017, background): “Another somewhat related RL approach is transfer learning, which aims to transfer learned behaviors from one domain to another [58], for example by initializing the parameters of policies in unsolved tasks [2], or by transferring skills across untrained robots [12].”
- Transfer Learning in Deep Reinforcement Learning: A Survey (2020, unknown): “Similar ideas of using UMA for inter-task mapping can also be found in [118] and [119].”