Bisimulation Metrics for Continuous Markov Decision Processes

By N. Ferns, Prakash Panangaden, Doina Precup. 2011

178 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Software, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Software

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “Deﬁnition 5 (Ferns et al. (2011)) .”
Monte Carlo Tree Search With Iteratively Refining State Abstractions (2021, background): “Perhaps the most natural notion of state similarity for abstraction reﬁning is a bisimulation metric [12, 11].”
Learning Invariant Representations for Reinforcement Learning without Reconstruction (2020, methodology): “Bisimulation metrics are a useful form of state abstraction, but prior methods to train distance functions either do not scale to pixel observations [6] (due to the max operator in Equation (3)), or were only designed for the (fixed) policy evaluation setting [1].”
Learning Domain Invariant Representations in Goal-conditioned Block MDPs (2021, methodology): “Previous work [6] tries to address these problems via model bisimulation metric [12].”
Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning (2022, background): “For this reason, Bisimulation Metrics (Ferns et al., 2011; Ferns and Precup, 2014; Castro, 2020) softens the concept of state partitions, and instead defines a pseudometric space (S, d), where a distance function d : S × S 7→ R≥0 measures the “behavioral similarity” between two states1.”
Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games (2021, background): “Motivated by bisimulation metric [8] to measure the similarity of two states in MDPs: i , P as j ; d , which considers both the immediate reward and the following transition dynamics, we want to build the metric to measure the similarity of two policies in a given Markov game through the…”
An automated measure of MDP similarity for transfer in reinforcement learning (2014, background): “One notable example are bisimulation metrics (Ferns et al. 2006; Ferns, Panangaden, and Precup 2011), which quantify differences between MDPs.”
Block Contextual MDPs for Continual Learning (2021, background): “Bisimulation metrics (Ferns et al., 2011) deﬁne a distance between states as follows: A nice property of this metric ˜ d is that difference in optimal value between two states is bounded by their distance as deﬁned by this metric.”
Offline Reinforcement Learning with Pseudometric Learning (2021, methodology): “To define a bonus b, we first learn a pseudometric d on the state-action space S ×A similarly to bisimulation metrics (Ferns et al., 2004; 2011; 2006), with the difference being that we are interested in pseudometrics in state-action space S × A rather than state space S.”
Sharing Experience in Multitask Reinforcement Learning (2019, background): “For continuous or large environments where experiences would be so sparse, it is potential to apply a confident metric to measure the similarity of states (acoss tasks) with respect to long-term behavior such as bisimulation metric [Ferns et al., 2011].”
Learning Markov State Abstractions for Deep Reinforcement Learning (2021, background): “Subsequent work on approximate MDP homomorphisms (Ravindran & Barto, 2004) and bisimulation metrics (Ferns et al., 2004, 2011) relaxed these strict assumptions and allowed ground states to have varying degrees of “bisimilarity.””
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, methodology): “Bisimulation metrics (Givan et al., 2003; Ferns et al., 2011) Since we are only interested in computing the coupling between the states in X and Y , the above formulation assumes that P X ( y ) = 0 for all y ∈ Y and P Y ( x ) = 0 for all x ∈ X . representation it The computation of d ∼ is expensive…”