DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning

By Rasool Fakoor, Pratik Chaudhari, Alex Smola. 2020

6 citations recorded by the source corpus.

This paper prescribes a suite of techniques for off-policy Reinforcement Learning (RL) that simplify the training process and reduce the sample complexity. First, we show that simple Deterministic Policy Gradient works remarkably well as long as the overestimation bias is controlled. This is contrast to existing literature which creates sophisticated off-policy techniques. Second, we pinpoint training instabilities, typical of off-policy algorithms, to the greedy policy update step; existing solutions such as delayed policy updates do not mitigate this issue. Third, we show that ideas in the propensity estimation literature can be used to importance-sample transitions from the replay buffer and selectively update the policy to prevent deterioration of performance. We make these claims using extensive experimentation on a set of challenging MuJoCo tasks. A short video of our results can be seen at this https URL .

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “While overestimation bias has been mainly studied in regard to function approximation error [13, 16, 23, 33, 58], Hasselt [22] shows that overestimation can also arise in tabular MDPs due to noise in the environment.”