Optimizing Average Reward Using Discounted Rewards

By Sham M. Kakade. 2001

46 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “The idea of compatible features motivates an elegant method for computing the natural policy gradient (Kakade, 2001a; Peters & Schaal, 2008).”
Preventing undesirable behavior of intelligent machines (2019, background): “Others have considered the problem of guaranteeing policy improvement with high confidence [204, 28], sometimes in the context of determining how the complexity of (approximately) solving MDPs grows with different MDP parameters, such as the number of possible states [205].”
On Ensuring that Intelligent Machines Are Well-Behaved (2017, background): “Others have considered the problem of guaranteeing policy improvement with high confidence in the context of determining how the complexity of (approximately) solving MDPs grows with different parameters of the MDP, like the number of possible states [Kakade, 2001].”
Metalearning and neuromodulation (2002, background): “It has been shown that the value function learned with a large γ tends to have a large variance, and thus learning with a smaller γ can result in biased, but less variable estimate of future rewards (Littman et al., 1995; Baxter & Bartlett, 2000; Kakade, 2001).”
On the sample complexity of reinforcement learning. (2003, methodology): “However, this chapter presents an anal ysis showing how the lack of exploration in gradient methods leads to an unreasonably (and arbitrarily) large variance in the estimates of the gradient direction (as discussed in Kakade and Langford [2002]).”
A Natural Policy Gradient (2001, background): “Although the previous sections argued that our choice is appropriate, we would like to understand how F relates to the Hessian r(2) ( ), which, as shown in [5], has the form: r(2) ( ) = X”
Bias in Natural Actor-Critic Algorithms (2014, background): “Kakade (2001) showed that ¯ J, which includes discounting in V θ , is the typical average reward objective function.”
Reinforcement learning of motor skills with policy gradients (2008, methodology): “This natural policy gradient approach was first suggested for reinforcement learning as the ‘average natural policy gradient’ in Kakade (2002), and subsequently shown to be the true natural policy gradient (Bagnell & Schneider, 2003; Peters, Vijayakumar, & Schaal, 2003).”