Compositional Imitation Learning: Explaining and executing one task at a time

By Thomas Kipf, Yujia Li, Hanjun Dai, Vinícius Flores Zambaldi, Edward Grefenstette, Pushmeet Kohli, Peter W. Battaglia. 2018

13 citations recorded by the source corpus.

We introduce a framework for Compositional Imitation Learning and Execution (CompILE) of hierarchically-structured behavior. CompILE learns reusable, variable-length segments of behavior from demonstration data using a novel unsupervised, fully-differentiable sequence segmentation module. These learned behaviors can then be re-composed and executed to perform new tasks. At training time, CompILE auto-encodes observed behavior into a sequence of latent codes, each corresponding to a variable-length segment in the input sequence. Once trained, our model generalizes to sequences of longer length and from environment instances not seen during training. We evaluate our model in a challenging 2D multi-task environment and show that CompILE can find correct task boundaries and event encodings in an unsupervised manner without requiring annotated demonstration data. Latent codes and associated behavior policies discovered by CompILE can be used by a hierarchical agent, where the high-level policy selects actions in the latent code space, and the low-level, task-specific policies are simply the learned decoders. We found that our agent could learn given only sparse rewards, where agents without task-specific policies struggle.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “Alternative designs, such as Poisson prior (Kipf et al., 2018) for latent space and Transformer (Vaswani et al.”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, methodology): “We compare to conventional model free RL (SAC, [10], w/ and w/o behavioral cloning initialization); learning priors over primitive actions not skills ("Flat Prior", similar to [31]); leveraging transferred skills without guidance via a learned prior ("w/o prior", representative of [22, 16, 29]).”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, methodology): “Kipf et al. (2019) propose a related method for video segmentation via generative modeling, and use it for hierarchical reinforcement learning.”
KeyIn: Discovering Subgoal Structure with Keyframe-based Video Prediction (2019, methodology): “In parallel to our work, Kipf et al. (2018) propose a similar method for sequence segmentation via variational inference with continuous relaxation of segment boundaries.”