Information asymmetry in KL-regularized RL

By Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojciech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, Nicolas Manfred Otto Heess. 2019

107 citations recorded by the source corpus.

Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning. We start from the KL regularized expected reward objective which introduces an additional component, a default policy. Instead of relying on a fixed default policy, we learn it from data. But crucially, we restrict the amount of information the default policy receives, forcing it to learn reusable behaviors that help the policy learn faster. We formalize this strategy and discuss connections to information bottleneck approaches and to the variational EM algorithm. We present empirical results in both discrete and continuous action domains and demonstrate that, for certain tasks, learning a default policy alongside the policy can significantly speed up and improve learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…on policy priors that are optimized to be different from the actual policy but so far mainly in the multi-task or transfer-learning setup, i.e. to share knowledge across or to transfer knowledge to new tasks (Teh et al., 2017; Galashov et al., 2019; Tirumala et al., 2019; Jaques et al., 2017).”
Multitask Soft Option Learning (2019, background): “This introduces an information-asymmetry between the task-specific master policy, and the shared options, allowing them to generalize well (Galashov et al., 2019).”
Regularized Hierarchical Policies for Compositional Transfer in Robotics (2019, unknown): “…on optimizing initial parameters [e.g. Finn et al., 2017], sharing models and parameters across tasks [e.g. Rusu et al., 2016, Teh et al., 2017, Galashov et al., 2018], data-sharing across tasks [e.g. Riedmiller et al., 2018, Andrychowicz et al., 2017] or related, auxiliary objectives…”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, methodology): “To retain the role of default policy as a regularizer, it has been explored to restrict its input (Galashov et al., 2019; Grau-Moya et al., 2019), param-eteric form (Czarnecki et al., 2018) or to share it across different contexts (Teh et al., 2017; Ghosh et al., 2018).”
Toward Robust Long Range Policy Transfer (2021, methodology): “Some methods intend to limit dependency between pretraining policy and task-specific information (Goyal et al. 2019; Galashov et al. 2019) by using information bottleneck (Alemi et al.”
Continual Adaptation for Efficient Machine Communication (2019, background): “Specifically, we consider a KL regularization term that explicitly minimizes the divergence between the captioning model’s output probabilities before and after finetuning for unseen images (Yu et al., 2013; Galashov et al., 2018).”
Regularization Matters in Policy Optimization (2019, background): “However, most of these regularizations are rather complicated (Galashov et al., 2019), specifically designed for certain algorithms (Parisi et al.”
Action and Perception as Divergence Minimization (2020, background): “Control as inference It is well known that RL can be formulated as KL minimization over inputs and actions (Todorov, 2008; Kappen et al., 2009; Rawlik et al., 2010; Ortega and Braun, 2011; Levine, 2018), as well as skills (Hausman et al., 2018; Tirumala et al., 2019; Galashov et al., 2019).”