Hierarchical Imitation and Reinforcement Learning

By Hoang Minh Le, Nan Jiang, Alekh Agarwal, Miroslav Dud'ik, Yisong Yue, Hal Daumé Iii. 2018

221 citations recorded by the source corpus.

We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the hierarchical structure of the underlying problem to integrate different modes of expert interaction. Our framework can incorporate different combinations of imitation learning (IL) and reinforcement learning (RL) at different levels, leading to dramatic reductions in both expert effort and cost of exploration. Using long-horizon benchmarks, including Montezuma's Revenge, we demonstrate that our approach can learn significantly faster than hierarchical RL, and be significantly more label-efficient than standard IL. We also theoretically analyze labeling cost for certain instantiations of our framework.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “ifferent goals [17, 18, 19, 20]. We introduce a novel form of goal relabeling and demonstrate its efﬁciency when applied to learning robust bi-level policies. A related idea is presented in Le et al. [27], where the authors assume that an expert provides labelled and segmented demonstrations at both”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “…adopted as a standard approach in robotics [Pomerleau, 1989, Schaal, 1997, Argall et al., 2009, Osa et al., 2017, Ye and Alterovitz, 2017, Aleotti and Caselli, 2006, Lawitzky et al., 2012, Torabi et al., 2018, Le et al., 2017, 2018] and many other areas such as playing games [Mnih et al., 2013].”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, background): “This approach was shown to be substantially more sample efﬁcient than conventional imitation learning on a very difﬁcult Atari game called Mon-tezuma’s revenge [Le et al. , 2018] .”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, methodology): “While the combination of demonstrations and reinforcement has been studied extensively in single task problems [20, 38, 29, 22], this combination is particularly important in meta-learning contexts where few-shot learning of new tasks is simply not possible without demonstrations.”
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning (2018, background): “There has been a growing interest in combining extrinsic sparse reward signals with imitation learning for guided exploration (Zhu et al., 2018; Kang et al., 2018; Le et al., 2018; Vecer´ık et al., 2017).”
Neural Modular Control for Embodied Question Answering (2018, background): “[6], and can be seen as an instantiation of the options framework [7, 8], wherein a global master policy proposes subgoals – to be achieved by local sub-policies – towards a downstream task objective [9–11].”
Learning Hierarchical Teaching Policies for Cooperative Agents (2019, background): “The paper presents HMAT, which utilizes the deep representation and HRL to transfer knowledge between agents in cooperative MARL.”
Composing Complex Skills by Learning Transition Policies (2018, background): “We also assume our primitive policies return a signal that indicates whether the execution should be terminated or not, similar to Kulkarni et al. (2016); Oh et al. (2017); Le et al. (2018).”
On Multi-Agent Learning in Team Sports Games (2019, background): “(Le et al., 2018) consider a hierarchical approach where the underlying low level actions are learned via RL whereas the high-level goals are picked up via IL from human demonstrations.”
Continuous Control with Action Quantization from Demonstrations (2021, background): “…from demonstrations together with a high-level controller that is either learned from demonstrations (Kroemer et al., 2015; Krishnan et al., 2017; Le et al., 2018; Ding et al., 2019; Lynch et al., 2020), or learned from interactions with the environment (Manschitz et al., 2015; Kipf et al.,…”
Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, methodology): “Hierarchical imitation learning methods reduce the sample complexity of standard imitation learning by performing data aggregation by organizing the action spaces in a hierarchy (Le et al., 2018).”
Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset (2019, background): “It has yet to be explored whether a learning agent can learn from this type of human demonstrated options, but results from hierarchical imitation learning (Le et al. 2018) indicate that this may indeed be possible.”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (2023, methodology): “Inspired by the concept of imitation learning in RL, which clones the behavior of positive demonstrations (Le et al., 2018), we propose to leverage negative samples to penalize the LM for imitating the mismatched target (i.e., value-aligned but incoherent).”