Using Bisimulation for Policy Transfer in MDPs

By Pablo Samuel Castro, Doina Precup. 2010

70 citations recorded by the source corpus.

Knowledge transfer has been suggested as a useful approach for solving large Markov Decision Processes. The main idea is to compute a decision-making policy in one environment and use it in a different environment, provided the two are ”close enough”. In this paper, we use bisimulation-style metrics (Ferns et al., 2004) to guide knowledge transfer. We propose algorithms that decide what actions to transfer from the policy computed on a small MDP task to a large task, given the bisimulation distance between states in the two tasks. We demonstrate the inherent ”pessimism” of bisimulation metrics and present variants of this metric aimed to overcome this pessimism, leading to improved action transfer. We also show that using this approach for transferring temporally extended actions (Sutton et al., 1999) is more successful than using it exclusively with primitive actions. We present theoretical guarantees on the quality of the transferred policy, as well as promising empirical results.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “…used for state aggregation (Ferns et al., 2004; Ruan et al., 2015), feature discovery (Comanici & Precup, 2011) and transfer learning between MDPs (Castro & Precup, 2010), but due to their high computational cost and poor compatibility with deep networks they have not been successfully applied to…”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Motivated by similar intuitions, research into bisimulation metrics has aimed to devise or learn similarity functions between states (Ferns et al., 2004; Castro and Precup, 2010).”
Deictic Image Maps: An Abstraction For Learning Pose Invariant Manipulation Policies (2018, background): “MDP Homomorphisms are related to other MDP abstraction methods [11, 12, 18].”
Invariant Causal Prediction for Block MDPs (2020, background): “These relations can be used to simplify the state space for tasks like policy transfer (Castro and Precup, 2010), and are intimately tied to state abstraction.”
Automatic Construction of Temporally Extended Actions for MDPs Using Bisimulation Metrics (2011, methodology): “An equivalence relation E is a bisimulation relation if for any s, t ∈ S, sEt implies that for all a ∈ A (i): R(s, a) = R(t, a); and (ii): for all C ∈ S/E, ∑ s′∈C P (s, a)(s ′) = ∑ s′∈C P (t, a)(s ′), where S/E is the set of all equivalence classes in S w.r.t. E.”
Scalable methods for computing state similarity in deterministic Markov Decision Processes (2019, methodology): “2006), policy transfer (Castro and Precup 2010), representation discovery (Ruan et al.”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, methodology): “We also contrast PSEs with bisimulation transfer (Castro & Precup, 2010), a tabular state-based transfer approach based on bisimulation metrics which does not do any learning and bisimulation preserving representations (Zhang et al., 2021), showing the advantages of PSM over a prevalent state…”
Hierarchical Approaches (2012, background): “Castro and Precup (2010) show that transferring abstract actions is more successful than just primitive actions using a variant of the bisimulation metric.”