Hyperparameter Selection for Imitation Learning

By L'eonard Hussenot, Marcin Andrychowicz, Damien Vincent, Robert Dadashi, Anton Raichuk, Lukasz Stafiniak, Sertan Girgin, Raphaël Marinier, Nikola Momchev, Sabela Ramos and 4 more. 2021

21 citations recorded by the source corpus.

We address the issue of tuning hyperparameters (HPs) for imitation learning algorithms in the context of continuous-control, when the underlying reward function of the demonstrating expert cannot be observed at any time. The vast literature in imitation learning mostly considers this reward function to be available for HP selection, but this is not a realistic setting. Indeed, would this reward function be available, it could then directly be used for policy training and imitation would not be necessary. To tackle this mostly ignored problem, we propose a number of possible proxies to the external reward. We evaluate them in an extensive empirical study (more than 10'000 agents across 9 environments) and make practical recommendations for selecting HPs. Our results show that while imitation learning algorithms are sensitive to HP choices, it is often possible to select good enough HPs through a proxy to the reward function.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Olivier Pietquin
2. Matthieu Geist
3. L'eonard Hussenot
4. Robert Dadashi
5. Damien Vincent
6. Sertan Girgin
7. Mingfei Sun
8. Anton Raichuk

In Computer Vision and Pattern Recognition

1. L'eonard Hussenot
2. Katja Hofmann
3. Anton Raichuk
4. Sam Devlin
5. Tim Pearce
6. Olivier Pietquin
7. Matthieu Geist
8. David Bignell

In Control and Systems Engineering

1. L'eonard Hussenot
2. Katja Hofmann
3. Anton Raichuk
4. Olivier Pietquin
5. Matthieu Geist
6. Sertan Girgin
7. Tabish Rashid
8. Damien Vincent

In Reinforcement Learning

1. Olivier Pietquin
2. Matthieu Geist
3. L'eonard Hussenot
4. Katja Hofmann
5. Anton Raichuk
6. Sam Devlin
7. David Bignell
8. Anssi Kanervisto

In Meta & Continual Learning

1. Katja Hofmann
2. Olivier Pietquin
3. Chelsea Finn
4. Sergey Levine

Citations in the corpus

• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, neutral): “Recently, Hussenot et al. [70] presented an empirical study on the importance of hyperparameter selection in imitation learning – it consists of an extensive evaluation of both offline and online imitation learning algorithms using modest-sized datasets (~10s of trajectories).”
• Continuous Control with Action Quantization from Demonstrations (2021, background): “There is no reward function and the notion of success is illdefined (Hussenot et al., 2021).”
• Imitating Human Behaviour with Diffusion Models (2023, methodology): “Finally, similarly to Dadashi et al. (2020); Hussenot et al. (2021), we compute the Wasserstein distance between the occupancy distributions as an evaluation criteria.”