An automated measure of MDP similarity for transfer in reinforcement learning

By Haitham Ammar, Eric Eaton, Matthew E. Taylor, Decebal Constantin Mocanu, Kurt Driessens, Gerhard Weiss, Karl Tuyls. 2014

103 citations recorded by the source corpus.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “In fact, some previous work showed that task similarity helps improve performance in the context of transfer learning [2, 3, 35], while some others claimed that task dissimilarity could help continual learning [13] although their discussion was more related to the permuted MNIST setting.”
Online Contrastive Divergence with Generative Replay: Experience Replay without Storing Data (2016, neutral): “…the similarity between data distributions in various domains (e.g., image quality assessment (Mocanu et al, 2014), Markov decision processes (Ammar et al, 2014)), could be used to perform anomaly detection directly on any wireless node if they would have available an online training…”
OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations (2021, background): “A Transferability metric can quantitatively reveal how easy it is to transfer knowledge learned from a source task to the target task [13, 4, 36, 24].”
Context-Aware Policy Reuse (2018, background): “policies to speed up the learning on a new target task [5, 13, 15, 25]. Many existing policy reuse approaches focus on finding a single best source policy for reuse, e.g., by measuring MDP similarity [2, 30], through online exploration using multi-armed bandit methods [21, 23], or via an optimism-un”
LEEP: A New Measure to Evaluate Transferability of Learned Representations (2020, background): “Transferability estimation (Eaton et al., 2008; Ammar et al., 2014; Sinapov et al., 2015) is the problem of quantitatively estimating how easy it is to transfer knowledge learned from one classification task to another.”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, background): “…structure, which can lead to faster learning and better generalization to unseen tasks from the same family (Taylor & Stone, 2009; Lazaric, 2012; Ammar et al., 2012; 2014; Parisotto et al., 2015; Borsa et al., 2016; Gupta et al., 2017; Andreas et al., 2017; Oh et al., 2017; Hessel et al., 2019).”
Adapting Auxiliary Losses Using Gradient Similarity (2018, methodology): “In RL, methods have been proposed to use the Markov decision process (MDP) similarity as a measure of task relatedness (Carroll and Seppi, 2005; Ammar et al., 2014).”
Measuring the Distance Between Finite Markov Decision Processes (2016, background): “Some work just measures the distance between dynamics of two MDPs, such as reward functions or transition models, and requires that the state-action spaces are the same [2, 6, 19].”
Structural Similarity for Improved Transfer in Reinforcement Learning (2022, unknown): “Works such as [1] and [17] propose metrics that require training models to extract useful information on source and target tasks.”
Learning Curriculum Policies for Reinforcement Learning (2018, background): “These approaches typically compute a similarity measure between the MDPs of the source and target task [2, 7], or learn a model of transferability that can be applied to novel source-target task pairs [11, 20].”