Exploiting Hierarchy for Learning and Transfer in KL-regularized RL

By Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, Nicolas Manfred Otto Heess. 2019

45 citations recorded by the source corpus.

As reinforcement learning agents are tasked with solving more challenging and diverse tasks, the ability to incorporate prior knowledge into the learning system and to exploit reusable structure in solution space is likely to become increasingly important. The KL-regularized expected reward objective constitutes one possible tool to this end. It introduces an additional component, a default or prior behavior, which can be learned alongside the policy and as such partially transforms the reinforcement learning problem into one of behavior modelling. In this work we consider the implications of this framework in cases where both the policy and default behavior are augmented with latent variables. We discuss how the resulting hierarchical structures can be used to implement different inductive biases and how their modularity can benefit transfer. Empirically we find that they can lead to faster learning and transfer on a range of continuous control tasks.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “This idea was introduced in earlier work in continual learning [46] (see also M.B. Ring, PhD thesis, University of Texas at Austin, 1994) and has shown promise with new research [47,48], but deserves further study at large scale.”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…on policy priors that are optimized to be different from the actual policy but so far mainly in the multi-task or transfer-learning setup, i.e. to share knowledge across or to transfer knowledge to new tasks (Teh et al., 2017; Galashov et al., 2019; Tirumala et al., 2019; Jaques et al., 2017).”
Multitask Soft Option Learning (2019, background): “ 2017) or desirable sub-goals (Tessler et al., 2017) in order to learn skills which transfer well between tasks. In contrast, our work aims at ﬁnding good termination states without such supervision. Tirumala et al. (2019) investigate the use of different priors for the higher-level policy while we ”
Regularized Hierarchical Policies for Compositional Transfer in Robotics (2019, unknown): “…been used for the discovery of behavior abstractions as part of an end-to-end reinforcement learning paradigm [e.g. Teh et al., 2017, Igl et al., 2019, Tirumala et al., 2019, Galashov et al., 2018] where the models act as learned inductive biases that induce the sharing of behavior across tasks.”
Action and Perception as Divergence Minimization (2020, methodology): “Control as inference It is well known that RL can be formulated as KL minimization over inputs and actions (Todorov, 2008; Kappen et al., 2009; Rawlik et al., 2010; Ortega and Braun, 2011; Levine, 2018), as well as skills (Hausman et al., 2018; Tirumala et al., 2019; Galashov et al., 2019).”