Task-Relevant Adversarial Imitation Learning

By Konrad Zolna, Scott E. Reed, Alexander Novikov, Sergio Gomez Colmenarejo, David M. Budden, Serkan Cabi, Misha Denil, Nando de Freitas, Ziyun Wang. 2019

65 citations recorded by the source corpus.

We show that a critical problem in adversarial imitation from high-dimensional sensory data is the tendency of discriminator networks to distinguish agent and expert behaviour using task-irrelevant features beyond the control of the agent. We analyze this problem in detail and propose a solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning (GAIL). Our proposed solution, Task-Relevant Adversarial Imitation Learning (TRAIL), uses a constrained optimization objective to overcome task-irrelevant features. Comprehensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators, where other agents such as behaviour cloning (BC), standard GAIL, improved GAIL variants including our newly proposed baselines, and Deterministic Policy Gradients from Demonstrations (DPGfD) fail to find solutions, even when the other agents have access to task reward.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Imitating Interactive Intelligence (2020, methodology): “We are not aware of extant work that has applied GAIL to observations this high-dimensional (see Li et al. (2017); Zolna et al. (2019) for applications of GAIL to simpler but still visual input), and, perhaps, for good reason.”
Shaking the foundations: delusions in sequence models for interaction and control (2021, background): “2The self-delusions discussed here are unrelated to the problem arising from observational confounding [2, 8, 25].”
A Generalist Agent (2022, methodology): “The DeepMind Manipulation Playground (Zolna et al., 2021) is a suite of MuJoCo based simulated robot tasks.”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, methodology): “…imitation learning method, however standard GAIL does not work in the following settings: 1) POMDPs (Gangwani et al., 2019; ˙Zołna et al., 2019), 2) from pixels (Li et al., 2017; Reed et al., 2018), 3) off policy (Kostrikov et al., 2018) and 4) with variable initial conditions (Zolna et al., 2019).”
Positive-Unlabeled Reward Learning (2019, methodology): “[13] to the inputs of all the discriminator models evaluated in the experiments.”