On the Role of Weight Sharing During Deep Option Learning

By Matthew Riemer, Ignacio Cases, Clemens Rosenbaum, Miao Liu, G. Tesauro. 2019

22 citations recorded by the source corpus.

The options framework is a popular approach for building temporally extended actions in reinforcement learning. In particular, the option-critic architecture provides general purpose policy gradient theorems for learning actions from scratch that are extended in time. However, past work makes the key assumption that each of the components of option-critic has independent parameters. In this work we note that while this key assumption of the policy gradient theorems of option-critic holds in the tabular case, it is always violated in practice for the deep function approximation setting. We thus reconsider this assumption and consider more general extensions of option-critic and hierarchical option-critic training that optimize for the full architecture with each update. It turns out that not assuming parameter independence challenges a belief in prior work that training the policy over options can be disentangled from the dynamics of the underlying options. In fact, learning can be sped up by focusing the policy over options on states where options are actually likely to terminate. We put our new algorithms to the test in application to sample efficient learning of Atari games, and demonstrate significantly improved stability and faster convergence when learning long options. 1

Subfields: Artificial Intelligence, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “While much of the work on learning options has focused on single task learning (Bacon et al., 2017; Machado et al., 2017a,b; Harb et al., 2018; Khetarpal et al., 2020a; Riemer et al., 2020; Klissarov and Precup, 2021), work on discovering options in a multi-task context has shown some promising pote”
Context-Specific Representation Abstraction for Deep Option Learning (2021, background): “There have been two major high-level approaches in the recent literature for achieving efficient HRL: the options framework to learn abstract skills (Sutton, Precup, and Singh 1999; Bacon, Harb, and Precup 2017; Omidshafiei et al. 2018; Riemer, Liu, and Tesauro 2018; Riemer et al. 2020) and goal-con”