Hierarchical Actor-Critic

By Andrew Levy, Robert W. Platt, Kate Saenko. 2017

82 citations recorded by the source corpus.

We present a novel approach to hierarchical reinforcement learning called Hierarchical Actor-Critic (HAC). HAC aims to make learning tasks with sparse binary rewards more efficient by enabling agents to learn how to break down tasks from scratch. The technique uses of a set of actor-critic networks that learn to decompose tasks into a hierarchy of subgoals. We demonstrate that HAC significantly improves sample efficiency in a series of tasks that involve sparse binary rewards and require behavior over a long time horizon.

Subfields: Artificial Intelligence, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “Lastly, we leverage implicit knowledge of the world acquired by π g during world graph discovery to the subsequent HRL training.”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “A particularly promising approach was proposed in Nachum et al. [15] and Levy et al. [16], using goal conditioned policies at multiple layers of hierarchy for RL.”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, positive): “HER+HRL Hierarchical Actor-Critic (Levy et al., 2017) showed some promising results in applying HER in Hierarchical RL setup.”
Visual Reinforcement Learning with Imagined Goals (2018, background): “[25] use goal relabeling for sparse rewards problems with known goal spaces, restricting the resampled goals to states encountered along that trajectory, since almost any other goal will have no reward signal.”
Learning Latent Plans from Play (2019, background): “Goal conditioned policies have been explored extensively in the literature for reinforcement learning [6, 15, 16, 17, 18, 19], as well as for control via inverse models [20, 21, 22, 23].”
Data-Efficient Hierarchical Reinforcement Learning (2018, methodology): “They are a natural choice for a generally applicable HRL method and have therefore appeared as components of other HRL methods [43, 22, 30, 24].”
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning (2018, background): “…goal-conditioned hierarchical designs have either learned representations naively end-to-end (Vezhnevets et al., 2017), or not learned them at all (Levy et al., 2017; Nachum et al., 2018), we take a principled approach to representation learning in hierarchical RL, translating a bound on…”
Hindsight policy gradients (2017, background): “In hierarchical reinforcement learning, goal-conditional policies may enable agents to plan using subgoals, which abstracts the details involved in lower-level decisions (Oh et al., 2017; Vezhnevets et al., 2017; Kulkarni et al., 2016; Levy et al., 2017).”
Learning Abstract Options (2018, background): “Recent work [11] also breaks down the problem over a temporal hierarchy, but like [32] is based on learning a latent goal representation that modulates the policy behavior as opposed to options.”
Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning? (2019, methodology): “…options framework (Precup, 2000) and goal-conditioned hierarchies (Nachum et al., 2018b), due to their impressive success in recent work (Frans et al., 2018; Levy et al., 2017; Nachum et al., 2018a; 2019), though an examination of other architectures is an important direction for future research.”
Hierarchical Policy Learning is Sensitive to Goal Space Design (2019, background): “Recently, Hierarchical Actor-Critic (HAC) (Levy et al., 2017) and HierQ (Levy et al., 2018) have examined combining HER and hierarchy.”
Hierarchical Deep Multiagent Reinforcement Learning (2018, methodology): “This is commonly seen in recent hierarchical Deep RL approaches such as Option-Critic (Bacon, Harb, and Precup 2017), FeUdal Networks (FUNs) (Vezhnevets et al. 2017) and Hierarchical Actor-Critic (HAC) (Levy, Jr., and Saenko 2017).”
Generalized Hindsight for Reinforcement Learning (2020, methodology): “Several hierarchical methods (Levy et al., 2017; Nachum et al., 2018) train a low-level policy to achieve subgoals and a higher-level controller to propose those subgoals.”
Composing Complex Skills by Learning Transition Policies (2018, background): “Unsupervised approaches to discover meta actions have been proposed (Schmidhuber, 1990; Daniel et al., 2016; Bacon et al., 2017; Vezhnevets et al., 2017; Dilokthanakul et al., 2017; Levy et al., 2017; Frans et al., 2018; Co-Reyes et al., 2018; Mao et al., 2018).”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, unknown): “But to keep with the hierarchical learning perspective, standard HRL methods have been extended to the deep learning context with various frameworks [113, 114, 115, 116, 117].”
Combining Optimal Control and Learning for Visual Navigation in Novel Environments (2019, background): “Our work is also related to hierarchical RL [3, 10, 27, 43], where a policy is factorized into sub-policies that are controlled via a master policy.”
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification (2020, background): “The option can be discovered unsupervisedly (Schmidhuber, 1990; Levy et al., 2017; Bacon et al., 2017; Nachum et al., 2018), meta-learned (Frans et al.”
Learning Navigation Subroutines from Egocentric Videos (2019, unknown): “Sub-policies and Options in Hierarchical RL. Hierarchical RL is an active area of research [29, 30, 2], with a number of recent papers (such as [31, 32]).”