Measuring the Distance Between Finite Markov Decision Processes
By Jinhua Song, Yang Gao, Hao Wang, Bo An. 2016
79 citations recorded by the source corpus.
Markov decision processes (MDPs) have been studied for many decades. Recent research in using transfer learning methods to solve MDPs has shown that knowledge learned from one MDP may be used to solve a similar MDP better. In this paper, we propose two metrics for measuring the distance between finite MDPs. Our metrics are based on the Hausdorff metric which measures the distance between two subsets of a metric space and the Kantorovich metric for measuring the distance between probabilistic distributions. Our metrics can be used to compute the distance between reinforcement learning tasks that are modeled as MDPs. The second contribution of this paper is that we apply the metrics to direct transfer learning by finding the similar source tasks. Our third contribution is that we propose two knowledge transfer methods which transfer value functions of the selected source tasks to the target task. Extensive experimental results show that our metrics are effective in finding similar tasks and significantly improve the performance of transfer learning with the transfer methods.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Amy Zhang
- Ashvin Nair
- Yang Gao
- Philippe Hansen-Estruch
- Y. Liu
- Chelsea Finn
- Pieter Abbeel
Citations in the corpus
- Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns (2019, methodology): “A commonly used technique for defin-ing MDP similarity is bisimulation metric [Hu et al. , 2015; Song et al. , 2016; Ferns et al. , 2004], which exactly captures the property of rewards and state transitions in an MDP.”
- Structural Similarity for Improved Transfer in Reinforcement Learning (2022, unknown): “L G ] 27 J u l 2022 this bisimulation metric to compare similarity between separate MDPs [18], but their metric requires that two MDPs be homogeneous, i.e., having equivalent state representations, a one-to-one correspondence between action spaces, and by satisfying a condition they refer to as…”
- Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, methodology): “Song et al. (2016) propose a metric between MDPs using the Kantorovich and Hausdorff metrics and use it for transferring value functions between source tasks and a target task.”