Deterministic Policy Gradient Algorithms

By David Silver, Guy Lever, Nicolas Manfred Otto Heess, Thomas Degris, Daan Wierstra, Martin A. Riedmiller. 2014

4,557 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Continuous control with deep reinforcement learning (2015, methodology): “Instead, here we used an actor-critic approach based on the DPG algorithm [3].”
Sample Efficient Actor-Critic with Experience Replay (2016, positive): “Policy gradient methods have been at the heart of significant advances in AI and robotics (Silver et al., 2014; Lillicrap et al., 2015; Silver et al., 2016; Levine et al., 2015; Mnih et al., 2016; Schulman et al., 2015a; Heess et al., 2015).”
Memory-based control with recurrent neural networks (2015, methodology): “We primarily consider the Deterministic Policy Gradient algorithm (DPG) [24], which is an off-policy policy gradient algorithm that has recently produced promising results on a broad range of difficult, highdimensional continuous control problems, including direct control from pixels [14].”
Learning Continuous Control Policies by Stochastic Value Gradients (2015, methodology): “Unlike DPG, instead of assuming a deterministic policy, SVG(0) estimates the derivative around the policy noise E p ( η ) Q a π θ (cid:12)(cid:12) η .”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “…although the contextual bandit problems investigated here are lower-dimensional (with 21-dimensional state spaces and 7-dimensional action spaces) than the bandit problem in (Silver et al., 2014) (with no state space and 10, 25 and 50-dimensional action spaces), they are nevertheless much harder.”
Gradient Estimation Using Stochastic Computation Graphs (2015, methodology): “This idea is exploited by the deterministic policy gradient algorithm [22].”
Programmable Agents (2017, methodology): “All of our agents are trained using Deterministic Policy Gradient [31, 21], where both the actor and the critic are Programmable Networks.”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “A particularly popular off-policy actor-critic method, DDPG (Lillicrap et al., 2015), which is a deep variant of the deterministic policy gradient (Silver et al., 2014) algorithm, uses a Q-function estimator to enable off-policy learning, and a deterministic actor that maximizes this Q-function.”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, methodology): “…the learning signal from the discriminator D ψ to the generator p θ p T | S q , instead of using the standard policy gradient as in [7], we consider a novel approach related to deterministic policy gradient (DPG) [11], which estimates the gradient as below: where the expectation in Eqn.”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “In actor-critic methods, the policy, known as the actor, can be updated through the deterministic policy gradient algorithm (Silver et al., 2014): a ] , the expected return when performing action a in state s and following π after, is known as the critic or the value function.”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “In an attempt to improve the applicability of Q-learning to continuous control, Silver et al. (2014) and Lillicrap et al. (2015) developed an oﬀ-policy algorithm DDPG, leading to promising results on continuous control environments.”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “…discounted rewards , where ρ ( θ ) is the distribution over trajectories τ = ( x 0 , u 0 , x 1 , u 1 , . . . ) induced by the current policy: DPG [33] is a policy gradient algorithm for continuous action spaces that improves the deterministic policy function π via backpropagation of the…”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Broadly, these methods can be understood as trading off bias (Sutton et al., 1999; Silver et al., 2014; Lillicrap et al., 2016; Gu et al., 2016) and variance (Precup, 2000; Munos et al., 2016).”
Reinforcement Learning with Deep Energy-Based Policies (2017, background): “We compare our algorithm to DDPG (Lillicrap et al., 2015), which has been shown to achieve better sample efﬁciency on the continuous control problems that we consider than other recent techniques such as REINFORCE (Williams, 1992), TRPO (Schulman et al., 2015a), and A3C (Mnih et al., 2016).”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “…such as Q-learning (Watkins & Dayan, 1992; Sutton et al., 1999; Mnih et al., 2015; Gu et al., 2016b) and off-policy actor-critic methods (Lever, 2014; Lillicrap et al., 2016), can instead use all samples, including off-policy samples, by adopting temporal difference learning with experience replay.”
PGQ: Combining policy gradient and Q-learning (2016, methodology): “In policy gradient techniques the policy is represented explicitly and we improve the policy by updating the parameters in the direction of the gradient of the performance (Sutton et al., 1999; Silver et al., 2014; Kakade, 2001).”
Distributed Prioritized Experience Replay (2018, methodology): “…it with a continuous-action policy gradient system based on DDPG (Lillicrap et al., 2016), an implementation of deterministic policy gradients Silver et al. (2014) also similar to older methods (Werbos, 1990; Prokhorov & Wunsch, 1997), and tested it on continuous control tasks from the…”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “For this reason, continuous domains are often tackled using actor-critic methods (Konda & Tsitsiklis, 1999; Hafner & Riedmiller, 2011; Silver et al., 2014; Lillicrap et al., 2016), where a separate parameterized “actor” policy π is learned in addition to the Qfunction or value function “critic,” suc”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “This approximate gradient is used as a starting point in many widely used deep reinforcement learning algorithms (Silver et al., 2014; Lillicrap et al., 2015; Wang et al., 2016; Gu et al., 2017b).”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Actor-critic methods (Sutton et al., 2000; Silver et al., 2014) and λ -weighted return estimation (Tesauro, 1995; Schulman et al., 2015b) replace the high variance Monte-Carlo return with an estimate based on the sampled return and a function approximator.”