Strategic Attentive Writer for Learning Macro-Actions

By Alexander Sasha Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John P. Agapiou, Koray Kavukcuoglu. 2016

179 citations recorded by the source corpus.

We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub- sequences by learning for how long the plan can be committed to - i.e. followed without re-planing. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro- actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.

Subfields: Artificial Intelligence, Computer Science Applications, Developmental and Educational Psychology, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Developmental and Educational Psychology

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Value Prediction Network (2017, background): “Although STRAW [36] can maintain a sequence of future actions using an external memory, it cannot explicitly perform planning by simulating future rewards/values.”
Towards Deep Symbolic Reinforcement Learning (2016, background): “Each of these shortcomings is an active area of research in the DRL community, where data efﬁcient learning [8, 9], transfer learning [10, 11, 12, 13], planning [14, 15], and transparency [16] are all hot topics.”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “…local policies that map states to primitive actions for achieving subgoals, and a global policy that initiates suitable subgoals in a sequence to achieve the ﬁnal goal of a task (Bacon & Precup, 2015; Kulkarni et al., 2016; Vezhnevets et al., 2016; Tessler et al., 2017; Andreas et al., 2017).”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, methodology): “Action repeat is a heuristic used in all successful agents (Mnih et al., 2015; 2016; Bellemare et al., 2016b; Vezhnevets et al., 2016).”
The Option-Critic Architecture (2016, background): “…but approaches that can be used naturally with continuous state and/or action spaces have only recently started to become feasible (Konidaris et al. 2011; Niekum 2013; Mann, Mannor, and Precup 2015; Mankowitz, Mann, and Mannor 2016; Kulkarni et al. 2016; Vezhnevets et al. 2016; Daniel et al. 2016).”
Modular Multitask Reinforcement Learning with Policy Sketches (2016, background): “Techniques for learning options with less guidance than the present work include Bacon & Precup (2015) and Vezhnevets et al. (2016), and other general hierarchical policy learners include Daniel et al. (2012), Bakker & Schmidhuber (2004) and Menache et al. (2002).”
Variational Intrinsic Control (2016, background): “…to option learning where the goal is to ﬁnd a small number of options that are useful for a particular task (Sutton et al., 1999; McGovern & Barto, 2001; Stolle & Precup, 2002; Silver & Ciosek, 2012; Kulkarni et al., 2016; Mankowitz et al., 2014; Vezhnevets et al., 2016; Bacon & Precup, 2015).”
Data-Efficient Hierarchical Reinforcement Learning (2018, background): “In practice, regularizers are essential to learn multiple effective and temporally abstracted sub-policies [2, 16, 42].”
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2016, background): “More recently, Mnih et al. (2016) propose a DRAW-like (Gregor et al., 2015) recurrent neural network architecture that can learn temporally extended macro actions.”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “Several works in DRL have attempted HRL by the use of one top-level policy that chooses between subpolicies, where the division of states or goals in to subpolicies is achieved either manually [1, 121, 54] or automatically [2, 129, 130].”
Learning and Transfer of Modulated Locomotor Controllers (2016, background): “Recent work [28] proposes an architecture for discovering temporally extended macro actions from scratch.”
Hybrid Reward Architecture for Reinforcement Learning (2017, background): “ls) 2,168 626 A3C (channels) 2,423 589 HRA 25,304 23,770 Results. Figure 5 shows the training curves; Table 1 shows the ﬁnal score after training. The best reported ﬁxed start score comes from STRAW (Vezhnevets et al., 2016); the best reported random start score comes from the Dueling network archit”