Reinforcement Learning in Continuous Time and Space

By Kenji Doya. 2000

1,078 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Benchmarking Deep Reinforcement Learning for Continuous Control (2016, methodology): “…1908; Donaldson, 1960; Widrow, 1964; Michie & Chambers, 1968), Cart-Pole Swing Up (Kimura & Kobayashi, 1999; Doya, 2000), Mountain Car (Moore, 1990), Acrobot Swing Up (DeJong & Spong, 1994; Murray & Hauser, 1991; Doya, 2000), and Double Inverted Pendulum Balancing (Furuta et al., 1978).”
Linear Off-Policy Actor-Critic (2012, background): “The second problem is a pendulum problem (Doya, 2000).”
Gaussian Processes for Data-Efficient Learning in Robotics and Control (2015, methodology): “Dynamics models were only learned in [18], [45], using RBF networks and multilayered perceptrons, respectively.”
Value-gradient learning (2012, methodology): “The cost function of equation 15 was derived to form an efficient greedy policy, by following the method of [7].”
Reinforcement Learning by Value Gradients (2008, methodology): “In many problems time can be treated as a continuous variable, i.e. t ∈ [0 , F ], as considered by Doya (2000) and Baird (1994).”
Learning CPG Sensory Feedback with Policy Gradient for Biped Locomotion for a Full-Body Humanoid (2005, methodology): “In the following section, we introduce the deﬁnition of the value function in continuous time and space to derive the temporal difference error (TD error)(Doya 2000).”
An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time (2013, methodology): “A greedy policy is only possible when the right-hand side of (7) is efﬁcient to compute, which is possible in the continuous time situations described by [24] or [7, section 2.2], and in the single network adaptive critic (SNAC) described by [18].”
TD_gamma: Re-evaluating Complex Backups in Temporal Difference Learning (2011, background): “One approach is to descend the gradient ∇θE(θ), assuming that the R (n) st are noisy samples of V (st ) and not a function of θ, as in the derivation of TD(λ) [7]:”
Dopamine: generalization and bonuses (2002, background): “The use of a sum over discrete time steps rather than an integral over continuous time steps is mainly for convenience (see Doya, 1999 for a fully continuous version).”
Meta-learning in Reinforcement Learning (2003, methodology): “We simulated the continuous-time version of TD learning, which uses the gradient of the learned value function for calculating a greedy policy (Doya, 2000).”
Robust Reinforcement Learning (2005, positive): “We can then derive learning rule for value function approximator [3] as Vi = rJ8(t)ei(t), where rJ denotes the learning rate.”
Model-based imitation learning by probabilistic trajectory matching (2013, methodology): “For instance, model-free methods based on trajectory sampling [27], [20] or model-based RL algorithms that learn forward models of the robot, and, subsequently, use them for predictions [12], [5], [23], [11], are suitable.”
Estimating the Maximum Expected Value in Continuous Reinforcement Learning Problems (2017, background): “Swing-up Pendulum A more complex scenario is represented by the continuous control problem analyzed in this section: the swing-up pendulum with limited torque (Doya 2000).”
Evolutionary Computation for Reinforcement Learning (2012, methodology): “Of course, actor-critic methods (Doya, 2000; Peters and Schaal, 2008) and other techniques (Gaskett et al, 1999; Millán et al, 2002; van Hasselt and Wiering, 2007) can also be used to make temporal-difference methods suitable for continuous action spaces.”
Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning (2016, background): “Online estimation has been given considerable attention in the field of reinforcement learning, particularly in engineering applications (Doya 2000; Kober and Peters 2012), with special consideration given to algorithms that provide fast updates in situations where new data accumulate multiple times”
Sim2Real Transfer for Deep Reinforcement Learning with Stochastic State Transition Delays (2020, background): “There exist several works that propose RL algorithms for continuous time and space [30, 31].”