Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

By Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, Jonathan Tompson. 2018

296 citations recorded by the source corpus.

We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework. The first problem is implicit bias present in the reward functions used in these algorithms. While these biases might work well for some environments, they can also lead to sub-optimal behavior in others. Secondly, even though these algorithms can learn from few expert demonstrations, they require a prohibitively large number of interactions with the environment in order to imitate the expert for many real-world applications. In order to address these issues, we propose a new algorithm called Discriminator-Actor-Critic that uses off-policy Reinforcement Learning to reduce policy-environment interaction sample complexity by an average factor of 10. Furthermore, since our reward function is designed to be unbiased, we can apply our algorithm to many problems without making any task-specific adjustments.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, background): “…first popularized in machine learning by Wasserstein generative-adversarial networks (Arjovsky et al., 2017), but also used in imitation learning (Kostrikov et al., 2018), cross-domain disentan- glement (Gonzalez-Garcia et al., 2018), and uncertainty estimation (van Amersfoort et al., 2020).”
Hyperparameter Selection for Imitation Learning (2021, methodology): “Both, AIL and PWIL use SAC (Haarnoja et al., 2018a) with an entropy constraint (Haarnoja et al., 2018b) to train the policy and optionally use absorbing states introduced in DAC (Kostrikov et al., 2018).”
The MAGICAL Benchmark for Robust Imitation (2020, methodology): “IL algorithms are commonly tested on Reinforcement Learning (RL) benchmark tasks, such as those from OpenAI Gym [37, 23, 27, 8].”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “In the rest of the paper, the GAIL variant studied uses TD3 as the direct RL method similarly to DAC (Kostrikov et al., 2019) with the original GAIL reward.”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “While in principle this would require importance sampling if using off-policy data from the replay buffer R, prior work has observed that adversarial IRL can drop the importance weights both in theory [10] and in practice [22].”
Positive-Unlabeled Reward Learning (2019, methodology): “Using GAIL with an off-policy learning method strictly speaking requires importance sampling correction [32], but we did not find this to be necessary.”
SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards (2019, background): “As pointed out in Kostrikov et al. (2019), this makes comparisons to alternative methods unfair.”
Primal Wasserstein Imitation Learning (2020, methodology): “…replacing the f -divergence by the Wasserstein distance through the Kantorovich-Rubinstein duality [Villani, 2008] leads to better training stability, which a number of methods in IL have leveraged [Li et al., 2017, Lacotte et al., 2019, Kostrikov et al., 2019, Xiao et al., 2019, Liu et al., 2020].”
What Matters for Adversarial Imitation Learning? (2021, methodology): “The configuration space is so wide that it covers the whole family of AIL algorithms, in particular, it mostly covers the setups from AIRL [17] and DAC [31].”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, methodology): “, 2018), 3) off policy (Kostrikov et al., 2018) and 4) with variable initial conditions (Zolna et al.”
Continuous Control with Action Quantization from Demonstrations (2021, result): “This results in a very similar algorithm as the one proposed by Kostrikov et al. (2019).”