Reinforcement learning in continuous time: advantage updating

By Leemon C. Baird III. 1994

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Taming the Noise in Reinforcement Learning via Soft Updates (2015, background): “An early approach to Q-learning in continuous noisy domains was to learn, instead of the value function, the advantage functionA(s, a) = Q(s, a)−V (s) [23].”
Deep learning in neural networks: An overview (2014, background): “ximate V(x t) only from local information r t;V(x k). Many variants of traditional RL exist (e.g., Barto et al., 1983; Watkins, 1989; Watkins and Dayan, 1992; Moore and Atkeson, 1993; Schwartz, 1993; Baird, 1994; Rummery and Niranjan, 1994; Singh, 1994; Baird, 1995; Kaelbling et al., 1995; Peng and ”
Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics (2012, background): “In Baird (1994), the advantage updating technique for sampled continuous-time system was proposed.”
On a Formal Model of Safe and Scalable Self-driving Cars (2017, background): “However, as noted in [1], this problem is not a property of any particular function approximation class, but rather, it is inherent in the definition of the Q function.”
Reinforcement Learning by Value Gradients (2008, methodology): “In continuous-time problem domains, Doya (2000) uses the value-gradient explicitly in the greedy policy, and Baird (1994) deﬁnes an “advantage” function that implicitly references the value-gradient.”
Multiple Forms of Value Learning and the Function of Dopamine (2009, background): “One version of this -called " advantage learning " (Baird, 1994, Dayan & Balleine 2002) defines the advantage A of action a in state s as ) ( ) , ( ) , ( s V a s Q a s A !”
Dynamic treatment regimes (2009, background): “In general, each Q * t (t = 1, 2) can be written as Q t = g * t + max at Q * t (g * t = Q * t − max at Q * t ). g * t is called the advantage function (Baird 1994) at the t th decision point.”
Adaptive optimal control for continuous-time linear systems based on policy iteration (2009, unknown): “A first reinforcement learning attempt to determine optimal controllers for continuous-time discrete-state systems was the advantage updating algorithm (Baird, 1994) which adapts discretetime reinforcement learning techniques to the case when the sampling time goes to zero.”
Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs (2020, methodology): “…RL. Previous methods using semi-Markov decision processes (SMDPs) [Howard, 1964]—including Q -learning [Bradtke and Duff, 1995], advantage updating [Baird, 1994], policy gradient [Munos, 2006], actor-critic [Doya, 2000]—extend the standard RL framework to continuous time, but all use relatively…”
Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adaptive Controllers (2012, methodology): “In another approach, Euler’s method can be used to discretize the continuous-time Bellman equation (86) (see [52]).”
Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey (2020, unknown): “Instead of Qθ (s, a), the advantage function [16] Aθ (s, a) = Qθ (s, a) − V πθ (s) is used in the”
Temporal supervised learning for inferring a dialog policy from example conversations (2014, background): “With the optimal value function Q∗, the advantage function [18] is defined by: A(s, a) := maxbQ ∗(s, b) − Q∗(s, a).”