Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information

By Arjun Sharma, Mohit Sharma, Nicholas Rhinehart, Kris M. Kitani. 2018

76 citations recorded by the source corpus.

The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their resulting state-action trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “There has a been a number of hierarchical imitation learning (HIL) approaches [23, 24, 25, 9, 26] that typically focus on extracting transition segments from the demonstrations.”
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables (2019, background): “Several prior approaches for multi-task imitation learning [21, 14, 34] propose to use unstructured demonstrations without knowing the task distribution, but they neither study quick generalization to new tasks nor provide a reward function.”
Modeling Long-horizon Tasks as Sequential Interaction Landscapes (2020, background): “To address this issue, there has a been a number of hierarchical imitation learning approaches [16, 53, 14, 29] that focus on segmenting long-horizon tasks into subcomponents.”
CompILE: Compositional Imitation Learning and Execution (2018, background): “Option discovery has also been addressed in the context of inverse reinforcement learning (IRL) using generative adversarial networks (GANs) (Goodfellow et al., 2014) to find structured policies that are close to demonstration sequences (Hausman et al., 2017; Sharma et al., 2018).”
Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation (2020, background): “[31] M.”
Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives (2023, methodology): “Based on [61], many works have been proposed, such as InfoGAIL [78], Directed-InfoGAIL [79], Co-GAIL [80], all of them achieve competitive results in their implement fields.”
Learning Robot Skills with Temporal Variational Inference (2020, background): “Hierarchical Policy Learning: Fox et al. (2017); Krishnan et al. (2017); Kipf et al. (2019); Sharma et al. (2019) also address the problem of learning options from demonstrations without supervision.”