What Matters for Adversarial Imitation Learning?

By Manu Orsini, Anton Raichuk, L'eonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, Marcin Andrychowicz. 2021

102 citations recorded by the source corpus.

Adversarial imitation learning has become a popular framework for imitation in continuous control. Over the years, several variations of its components were proposed to enhance the performance of the learned policies as well as the sample complexity of the algorithm. In practice, these choices are rarely tested all together in rigorous empirical studies. It is therefore difficult to discuss and understand what choices, among the high-level algorithmic options as well as low-level implementation details, matter. To tackle this issue, we implement more than 50 of these choices in a generic adversarial imitation learning framework and investigate their impacts in a large-scale study (>500k trained agents) with both synthetic and human-generated demonstrations. While many of our findings confirm common practices, some of them are surprising or even contradict prior work. In particular, our results suggest that artificial demonstrations are not a good proxy for human data and that the very common practice of evaluating imitation algorithms only with synthetic demonstrations may lead to algorithms which perform poorly in the more realistic scenarios with human demonstrations.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Continuous Control with Action Quantization from Demonstrations (2021, result): “Notice that our results corroborate previous work (Orsini et al., 2021) that showed poor performance of GAIL on human demonstrations after 1M steps.”
Imitating Human Behaviour with Diffusion Models (2023, background): “However, for applications requiring cloning of human behaviour, which contains diverse trajectories and multimodality at decision points, simple models may not be expressive enough to capture the full range and ﬁdelity of behaviours (Orsini et al., 2021).”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “Over time, various enhancements have been proposed to GAIL, aiming to enhance its sample efﬁciency, scalability, and robustness [76], including modiﬁcations to the discriminator’s loss function [77] and transitioning from the on-policy to off-policy agents [78].”