The Option-Critic Architecture

By Pierre-Luc Bacon, Jean Harb, Doina Precup. 2016

1,311 citations recorded by the source corpus.

Temporal abstraction is key to scaling up learning and planning in reinforcement learning. While planning with temporally extended actions is well understood, creating such abstractions autonomously from data has remained challenging.We tackle this problem in the framework of options [Sutton,Precup and Singh, 1999; Precup, 2000]. We derive policy gradient theorems for options and propose a new option-critic architecture capable of learning both the internal policies and the termination conditions of options, in tandem with the policy over options, and without the need to provide any additional rewards or subgoals. Experimental results in both discrete and continuous environments showcase the flexibility and efficiency of the framework.

Subfields: Artificial Intelligence, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Finally, it is often useful to learn multiple general value functions (Sutton et al., 2011; Mankowitz et al., 2018; Lample & Chaplot, 2016; Mirowski et al., 2017; Jaderberg et al., 2017b) or options (Sutton et al., 1999; Bacon et al., 2017) from a single stream of experience.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “d G wis more favorable over V rand. 6 Related Works Pivotal state discovery is related to unsupervised sequence segmentation [17, 47, 76, 13, 16] and option or sub-task discovery in the context of RL [48, 6, 72, 30, 56, 53, 52]. Among them, both [52] and [76] employ sequential variational models to ”
Independently Controllable Features (2017, background): “There are many more works that go in this direction, such as (deep) successor feature representations (Dayan, 1993; Kulkarni et al., 2016) or the options framework (Sutton et al., 1999; Precup, 2000) when used in conjunction with neural networks (Bacon et al., 2016).”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “…local policies that map states to primitive actions for achieving subgoals, and a global policy that initiates suitable subgoals in a sequence to achieve the ﬁnal goal of a task (Bacon & Precup, 2015; Kulkarni et al., 2016; Vezhnevets et al., 2016; Tessler et al., 2017; Andreas et al., 2017).”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “In high-dimensional lifelong learning settings (E.g. Minecraft), learning from visual experiences provides a potential solution to learning reusable skills.”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “This demonstrates potential for our approach in a hierarchical RL setting, with links to options (Bacon, Harb, and Precup 2017).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, methodology): “While much of the work on learning options has focused on single task learning (Bacon, Harb, and Precup, 2017; Machado, Bellemare, and Bowling, 2017a; Machado, Rosenbaum, Guo, Liu, Tesauro, and Campbell, 2017b; Harb, Bacon, Klissarov, and Precup, 2018; Khetarpal, Klissarov, Chevalier-Boisvert,…”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Recent work has investigated deep hierarchical agents [1, 48, 16, 31], building on methods that explicitly design a hierarchy over actions [32, 12], or utilize options [43, 35].”
Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, background): “Options formalize the notion of temporally abstraction actions in RL. Interestingly, generic architectures designed for shallow (Bacon et al., 2017) or deep (Riemer et al., 2018) hierarchies of options in essence learn very complex patterns of weight sharing over time.”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, background): “[21], which recovers subtasks from demonstrations in the spirit of hierarchical RL [39, 1].”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “Solutions like the options framework [6, 10], HAM [11], max-Q [12], and feudal networks [13, 14] present promising algorithmic frameworks for HRL.”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “While some works (Bacon et al., 2017; Stolle & Precup, 2002; Peng et al., 2019) learn a discrete set of lower-level policies, each behaving as a primitive skill, other works (Vezhn-evets et al., 2017; Nachum et al., 2018b; 2019a; Hausman et al., 2018) learn a continuous space of primitive skills…”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “Learned skills can be represented as sub-policies in the form of options [33, 1], as subgoal setter and reacher functions [9, 21] or discrete primitive libraries [27, 17].”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “In hierarchical MARL, different subtasks are chosen concurrently by all agents, whereas only a single subtask is chosen for each segment in single-agent hierarchical RL [4, 41].”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, background): “However, unlike in options learning, where a meta-controller is tasked with choosing the goal that a controller is trained to master (Bacon, Harb, and Precup 2017), our approach requires an observer capable of identifying which strategy was performed.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “, Bacon et al. (2017); Heess et al. (2016); Dayan & Hinton (1993); Frans et al.”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, methodology): “The experimental results for Option-Critic on 4 ATARI (Bacon et al., 2017) games show scores similar those from a ﬂat DQN (Mnih et al., 2015) baseline agent.”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, methodology): “Techniques for learning options with less guidance than the present work include Bacon & Precup (2015) and Vezhn-evets et al. (2016), and other general hierarchical policy learners include Daniel et al. (2012), Bakker & Schmidhu-ber (2004) and Menache et al. (2002).”
Variational Intrinsic Control (2016, background): “…to option learning where the goal is to ﬁnd a small number of options that are useful for a particular task (Sutton et al., 1999; McGovern & Barto, 2001; Stolle & Precup, 2002; Silver & Ciosek, 2012; Kulkarni et al., 2016; Mankowitz et al., 2014; Vezhnevets et al., 2016; Bacon & Precup, 2015).”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “…control (Lillicrap et al. 2016; Duan et al. 2016), classic video-games such as Atari (Mnih et al. 2015; Hessel et al. 2018; Gruslys et al. 2018; Schulman et al. 2015; Schulman et al. 2017; Bacon, Harb, and Precup 2017), and 3D ﬁrst person environments (Mnih et al. 2016; Jaderberg et al. 2016).”