Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

By Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, Josh Tenenbaum. 2016

1,297 citations recorded by the source corpus.

Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for its own sake rather than to directly solve problems. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical value functions, operating at different temporal scales, with intrinsically motivated deep reinforcement learning. A top-level value function learns a policy over intrinsic goals, and a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse, delayed feedback: (1) a complex discrete stochastic decision process, and (2) the classic ATARI game `Montezuma's Revenge'.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

What can the brain teach us about building artificial intelligence? (2016, background): “Deep reinforcement learning is only just starting to address intrinsically motivated learning (Kulkarni et al., 2016; Mohamed & Rezende, 2015).”
Programmable Agents (2017, background): “Compositionality has become a topic of great interest in machine learning, robotics and cognitive science [20, 30, 19, 9, 35, 26], and it has been central to research in language [32, 25, 33, 36].”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “These challenges may be approachable by models that exploit hierarchy [8, 26], modularity [4, 16], and structured reasoning and planning [6].”
Concrete Problems in AI Safety (2016, background): “Hierarchical RL seems a particularly promising approach to oversight, especially given the potential promise of combining ideas from hierarchical RL with neural network function approximators [84].”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Among successful applications of HRL we highlight h-DQN (Kulkarni et al. 2016a) and Feudal Networks (Vezhnevets et al.”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “However, existing approaches either require hand-crafted object representations [20] or have not been shown to capture fine-grained representations over long temporal horizons [14].”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, background): “Others have shown benefits of intrinsic motivation for learning RL tasks (Eysenbach et al., 2018; Kulkarni et al., 2016; Pathak et al., 2017; Song et al., 2019), though the connections to policy robustness and object-based representations are not explored in these works.”
Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning (2017, background): “…local policies that map states to primitive actions for achieving subgoals, and a global policy that initiates suitable subgoals in a sequence to achieve the ﬁnal goal of a task (Bacon & Precup, 2015; Kulkarni et al., 2016; Vezhnevets et al., 2016; Tessler et al., 2017; Andreas et al., 2017).”
Graying the black box: Understanding DQNs (2016, methodology): “The ability to automatically detect sub-goals and skills can also be used in the future as a component of an hierarchical DRL algorithm (Tessler et al., 2017; Kulkarni et al., 2016).”
A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “There are deep learning approaches that perform sub-goal learning (Rusu et al. 2016; Kulkarni et al. 2016), yet these approaches rely on providing the task or sub-goal to the agent, prior to making a decision.”
End-to-End Driving Via Conditional Imitation Learning (2017, background): “Hierarchical approaches have also been combined with deep learning and applied to raw sensory input [20].”
Learning to Act by Predicting the Future (2016, methodology): “We have compared the presented approach to three deep RL methods: DQN (Mnih et al., 2015), A3C (Mnih et al., 2016), and DSR (Kulkarni et al., 2016b).”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “However, HRL methods have traditionally struggled due to various practical challenges such as exploration [5], skill segmentation [6] and reward deﬁ-nition [7].”
Exploration by Random Network Distillation (2018, background): “Special access to the underlying RAM state can also be used to improve exploration by using it to hand-craft exploration bonuses (Kulkarni et al., 2016; Tang et al., 2017; Stanton & Clune, 2018).”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “…2018; Nair et al., 2018; Gulcehre et al., 2020), variance reduction (Wang et al., 2016; Schulman et al., 2017; Espeholt et al., 2018), hierarchy (Kulkarni et al., 2016; Vezhnevets et al., 2017) or meta-learning (Al-Shedivat et al., 2017; Finn et al., 2017; Xu et al., 2018), to name a few…”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “Recent work built on deep reinforcement learning [24, 30] to demonstrate hierarchical single-agent RL in high-dimensional continuous state spaces, using predefined subgoals [16], end-to-end learning of options [4], and latent directional subgoals [41] in a two-level hierarchy.”
Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces (2019, methodology): “To address the problem we mentioned in previous section, we propose another novel algorithm Deep MAHHQN, as inspired by Hierarchical Learning [ Kulkarni et al. , 2016; Tang et al. , 2018 ] .”