Evaluation of Policy Gradient Methods and Variants on the Cart-Pole Benchmark

By Martin A. Riedmiller, Jan Peters, Stefan Schaal. 2007

92 citations recorded by the source corpus.

Subfields: Control and Systems Engineering, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Management Science and Operations Research

Citations in the corpus

Benchmarks for Deep Off-Policy Evaluation (2021, background): “Outside of OPE, standardized benchmark suites have led to considerable standardization and progress in RL (Stone & Sutton, 2001; Dutech et al., 2005; Riedmiller et al., 2007).”
RE-EVALUATE: Reproducibility in Evaluating Reinforcement Learning Algorithms (2018, background): “We emphasize that authors should release careful details about evaluation schemes used in their work, which can often lead to new contributions [21, 22].”
A Survey on Machine-Learning Techniques in Cognitive Radios (2013, methodology): “Instead of perturbing the parameter θk of a deterministic policy u = π(x) (with u being the action and x being the state), the VPG approach assumes a stochastic policy u ∼ π(u|x) and obtains an unbiased gradient estimate [120].”
Neuroevolution strategies for episodic reinforcement learning (2009, methodology): “The adaptation of the global step size and the covariance matrix in the CMA-NeuroES resembles learning the (Fisher) metric in natural PGMs [28,36,34].”
Policy Gradients with Parameter-Based Exploration for Control (2008, background): “However, recent results [11] indicate that this drawback was overestimated in the past.”
On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient (2010, methodology): “Following the formulation given in [24], our control input is drawn from the policy u ∼ N(K>x, σ), with state x = [x, ẋ, θ, θ̇] and policy parameters K = [K1, K2, K3, K4, σ].”
Reinforcement learning in robotic applications: a comprehensive survey (2021, background): “However, the estimated gradient has larger variance and calculated gradient have no knowledge of past estimates, a major disadvantage of actor-methods (Riedmiller et al. 2007; Peters et al. 2010; Sutton et al. 2000).”
Approximate Inference and Stochastic Optimal Control (2010, methodology): “2 Results We demonstrate the applicability of LSΨ on a pole on cart task [28], which has been repeatedly used as a benchmark in reinforcement learning [21, 23].”
Reinforcement learning in feedback control (2011, methodology): “…efficient RL methods, one also observes a growing number of successful control applications, e.g. helicopter control (Ng et al. 2004), car driving (Riedmiller et al. 2007a), learning of robot behaviours (Peters and Schaal 2006; El-Fakdi and Carreras 2008), control of soccer robots (Riedmiller et…”
Ieee Transactions on Systems, Man and Cybernetics—part C: Applications and Reviews 1 a Survey of Actor-critic Reinforcement Learning: Standard and Natural Policy Gradients (unknown): “Convergence is obtained if the estimated gradients are unbiased and the learning rates α a,k satisfy [7], [38] ∞ k=0 α a,k = ∞ ∞ k=0 α 2 a,k ∞ A drawback of the actor-only approach is that the estimated gradient may have a large variance [19], [43] .”
Policy Improvement : Between Black-Box Optimization and Episodic Reinforcement Learning (2013, background): “Riedmiller et al. (2007) provides a concise yet clear explanation how to derive (2).”
Parameter-exploring policy gradients (2010, unknown): “Additionally, recent results Riedmiller et al. (2007) indicate that this drawback was overestimated in the past.”
Exploring parameter space in reinforcement learning (2010, background): “The first scenario is the extended pole balancing benchmark as described in [17].”
Online Model Learning Algorithms for Actor-Critic Control (2015, background): “A drawback of the actor-only approach is that the estimated gradient may have a large variance (Riedmiller et al., 2007; Sutton et al., 2000).”