Model-Free reinforcement learning with continuous action in practice

By Thomas Degris, Patrick M. Pilarski, Richard S. Sutton. 2012

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

MaskGAN: Better Text Generation via Filling in the ______ (2018, background): “…the action a t is the token chosen by the generator a t ≡ ˆ x t and the state s t are the current tokens produced up to that point This approach is an actor-critic architecture where G determines the policy π ( s t ) and the baseline b t is the critic (Sutton & Barto, 1998; Degris et al., 2012).”
Asynchronous Methods for Deep Reinforcement Learning (2016, background): “This approach can be viewed as an actor-critic architecture where the policy π is the actor and the baseline b t is the critic (Sutton & Barto, 1998; Degris et al., 2012).”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “4 For λ a = 0 , the emphatic weighting is simply i , and the gradient with this weighting reduces to the regular off-policy actor critic update [Degris et al., 2012b].”
Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation (2018, positive): “ent denoted as ∇θJ(πθ)= Es∼ρπθ ,a∈π θ [∇θloдπθ(a|s)Qw(s,a)]. In an off-policy setting, actor-critic estimates the value function of πθ(a|s)by averaging the state distribution of behavior policy β(a|s)[11]. Instead of considering the stochastic policy πθ(s|a), the deterministic policy gradient (DPG) ”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, methodology): “A body of literature in reinforcement learning focuses on reducing the variance of (10) by using sophisticated techniques such as actor-critique methods [30, 9].”
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, methodology): “There are many natural actor-critics, including Natural policy gradient utilizing the Temporal Differences (NTD) algorithm [99], Natural Actor-Critic using LSTD-Q(λ) (NAC-LSTD) [100], Episodic Natural Actor-Critic (eNAC) [100], Natural Actor-Critic using Sarsa(λ) (NAC-Sarsa) [101], Incremental Natur”
A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers (2018, background): “actorcritic, to directly learn the policy without resorting to a value function [18,19].”
Bias in Natural Actor-Critic Algorithms (2014, methodology): “We argued that they, and INAC, act more like average reward natural actor-critics that do not properly account for how changes to θ change the expected return via d θ .”
Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning (2017, background): “the actor and the estimate of the state-value function Vv(st) implies to the critic [45, 46].”
RLLib: C++ Library to Predict, Control, and Represent Learnable Knowledge Using On/Off Policy Reinforcement Learning (2015, methodology): “SP Aac and CG OffPac represent swinging pendulum and continuous grid-world problems solved using average [4] and off-policy [5] actorcritic algorithms.”
Deterministic Policy Gradient Algorithms (2014, methodology): “…in favour of policy evaluation algorithms that estimate the value function more ef-ﬁciently by temporal-difference learning (Bhatnagar et al., 2007; Degris et al., 2012b; Peters et al., 2005); indeed if both i) and ii) are satisﬁed then the overall algorithm is equivalent to not using a critic at…”
OFFER: Off-Environment Reinforcement Learning (2017, methodology): “Policy gradient (PG) methods (Sutton et al. 2000; Silver et al. 2014; Degris, Pilarski, and Sutton 2012; Deisenroth et al. 2013) use gradient ascent to directly optimise θ. PG methods are appealing in many settings because they cope well with continuous action spaces.”
Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation (2019, background): “Techniques such as policy gradient (Williams, 1992) and actor-critic (Sutton and Barto, 2018; Degris et al., 2012) are thus required to find an unbiased estimation of the gradient to optimize the model.”
DEVELOPING A PREDICTIVE APPROACH TO KNOWLEDGE (2015, methodology): “Several studies reported good results when the secondary weights are learned slowly, using a small value for αh compared to α (Sutton et al., 2009; Degris et al., 2012; Hackman, 2013).”