Primal Wasserstein Imitation Learning

By Robert Dadashi, L'eonard Hussenot, Matthieu Geist, Olivier Pietquin. 2020

157 citations recorded by the source corpus.

Imitation Learning (IL) methods seek to match the behavior of an agent with that of an expert. In the present work, we propose a new IL method based on a conceptually simple algorithm: Primal Wasserstein Imitation Learning (PWIL), which ties to the primal form of the Wasserstein distance between the expert and the agent state-action distributions. We present a reward function which is derived offline, as opposed to recent adversarial IL algorithms that learn a reward function through interactions with the environment, and which requires little fine-tuning. We show that we can recover expert behavior on a variety of continuous control tasks of the MuJoCo domain in a sample efficient manner in terms of agent interactions and of expert interactions with the environment. Finally, we show that the behavior of the agent we train matches the behavior of the expert with the Wasserstein distance, rather than the commonly used proxy of performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Hyperparameter Selection for Imitation Learning (2021, methodology): “Primal Wasserstein Imitation Learning (PWIL, Dadashi et al. (2020)) uses an RL algorithm to minimize a greedy upper bound on the Wasserstein distance between the state-action distributions of the expert and the agent.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, neutral): “PWIL Continuous Actions Primal Wasserstein Imitation Learning (Dadashi et al., 2021) is a non-adversarial imitation learning method that aims at minimizing the primal formulation of the Wasserstein distance between the state-action occupancy measure of the agent and the one of the expert.”
Offline Reinforcement Learning with Pseudometric Learning (2021, background): “This should not come as a surprise since the Euclidean distance has been shown to be an effective measure of similarity in some continuous control tasks (Dadashi et al., 2021).”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “It spans from low data regime (imitation from a single subsampled trajectory [10]) to high data regime (ORL from large datasets [5]), from having access to states, actions and rewards [11] to having only access to observations [43].”
Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap (2021, background): “Practically, AdVIL bears similarity to the Wasserstein GAN (WGAN) framework (Arjovsky et al. 2017), though we consider an IPM rather than the more restricted Wasserstein distances.”
Continuous Control with Action Quantization from Demonstrations (2021, background): “A number of existing works (Ho & Ermon, 2016; Ghasemipour et al., 2019; Dadashi et al., 2021) cast the problem into matching the state distributions of the agent and of the expert.”
Imitating Human Behaviour with Diffusion Models (2023, methodology): “Finally, similarly to Dadashi et al. (2020); Hussenot et al. (2021), we compute the Wasserstein distance between the occupancy distributions as an evaluation criteria.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “To address these challenges, approaches like primal Wasserstein IL (PWIL) [79] have emerged.”