Decoupling Dynamics and Reward for Transfer Learning

By Amy Zhang, Harsh Satija, Joelle Pineau. 2018

80 citations recorded by the source corpus.

Current reinforcement learning (RL) methods can successfully learn single tasks but often generalize poorly to modest perturbations in task domain or training procedure. In this work, we present a decoupled learning strategy for RL that creates a shared representation space where knowledge can be robustly transferred. We separate learning the task representation, the forward dynamics, the inverse dynamics and the reward function of the domain, and show that this decoupling improves performance within the task, transfers well to changes in dynamics and reward, and can be effectively used for online planning. Empirical results show good performance in both continuous and discrete RL domains.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “With rapid progress in deep RL, recent work (Zhang, Satija, and Pineau, 2018b; François-Lavet, Bengio, Precup, and Pineau, 2019) has focused on building an abstract representation with a low-dimensional representation of relevant features for non-stationary settings.”
State Representation Learning for Control: An Overview (2018, methodology): “Conversely, [Zhang et al., 2018] shows in an ablation study that using an inverse model (along with a forward model and an auto-encoder) is the factor that contributes the most to learning a good state representation.”
Direct Policy Transfer via Hidden Parameter Markov Decision Processes (2018, background): “Zhang et al. (2018) decouples the dynamics and rewards when learning a task and is thus capable of transferring to problems where either the dynamics and rewards mechanism may change.”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “Works on meta-RL and transfer learning focus on learning representations over tasks and use them to solve new and previously unseen tasks [Doshi-Velez and Konidaris, 2016, Hausman et al., 2018, Zhang et al., 2018, Rakelly et al., 2019, Zintgraf et al., 2019].”
S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning (2018, methodology): “, 2017) combine both inverse and forward models and (Zhang et al., 2018) additionally integrate an auto-encoder.”
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images (2019, methodology): “…the representation is ﬁrst trained via the au-toencoder, and then either with a policy learned on top of the ﬁxed representation (Lange & Riedmiller, 2010; Munk et al., 2016; Higgins et al., 2017b; Zhang et al., 2018; Nair et al., 2018), or with planning (Mattner et al., 2012; Finn et al., 2015).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “This can be done for instance with an encoder-decoder architecture variant (Higgins et al., 2017; Zhang et al., 2018b).”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “A model-based approach along this line is [125], which learns a model to map the state observation s to a latent-representation z .”
Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials (2020, background): “(Zhang, Satija, and Pineau 2018) also learns and plans in a latent space with learned dynamics and reward models, and explores transfer using prior learned encoders on slightly perturbed tasks.”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, methodology): “Zhang et al. (2018b) use separate dynamics and reward modules to learn a task embedding.”
Combined Reinforcement Learning via Abstract Representations (2018, background): “One approach is to ﬁrst infer a factorized set of generative factors from the observations (e.g., with an encoder-decoder architecture variant (Zhang, Satija, and Pineau 2018), variational auto-encoder (Higgins et al. 2017) or using for instance t-SNE (Maaten and Hinton 2008)).”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, methodology): “Recent advances in deep reinforcement learning (RL) have largely relied on model-free approaches, demonstrating strong performance on a variety of domains (Silver et al., 2016; Mnih et al., 2013; Kempka et al., 2016; Zhang et al., 2018c).”
Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization (2021, background): “…agents need to ﬁnd structure in the world to tackle new tasks by decomposing them in smaller sub-problems through function composition (Grifﬁths et al. 2019) or by extracting meaningful information in the form of abstract concepts (Zhang, Satija, and Pineau 2018; Francois-Lavet et al. 2019).”
Learning Markov State Abstractions for Deep Reinforcement Learning (2021, background): “…from simulation to real-world problems (Christiano et al., 2016), enabling effective robot motion planning (Agrawal et al., 2016), defining intrinsic reward bonuses for exploration (Pathak et al., 2017; Choi et al., 2019), and decoupling representation learning from rewards (Zhang et al., 2018).”
Explainability in Deep Reinforcement Learning (2020, background): “Thus, the low-level agent will have a succession of sub-goals (e.g. multiple actions that the robotic arm must perform such as moving or opening its pinch) that will ultimately lead to the achievement of the high-level goal (i.e. grasping the red ball).”
Learning Actionable Representations with Goal-Conditioned Policies (2018, methodology): “In the context of RL and control, representation learning methods have been used for many down-stream applications (Lesort et al., 2018), including representing value functions (Barreto et al., 2016) and building models (Watter et al., 2015; Assael et al., 2015; Zhang et al., 2018b).”