Particle Value Functions

By Chris J. Maddison, Dieterich Lawson, George Tucker, Nicolas Manfred Otto Heess, Arnaud Doucet, Andriy Mnih, Yee Whye Teh. 2017

15 citations recorded by the source corpus.

The policy gradients of the expected return objective can react slowly to rare rewards. Yet, in some cases agents may wish to emphasize the low or high returns regardless of their probability. Borrowing from the economics and control literature, we review the risk-sensitive value function that arises from an exponential utility and illustrate its effects on an example. This risk-sensitive value function is not always applicable to reinforcement learning problems, so we introduce the particle value function defined by a particle filter over the distributions of an agent's experience, which bounds the risk-sensitive one. We illustrate the benefit of the policy gradients of this objective in Cliffworld.

Subfields: Economics and Econometrics, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Management Science and Operations Research

Citations in the corpus

Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “(1) For β < 0 this utility function is risk-averse and has been studied extensively in the literature on safety and risk-averse reinforcement learning [35, 83, 52, 53, 48, 50].”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “Many previous studies on risk-sensitive RL adopt the utility function approach (Howard & Matheson, 1972; Marcus et al., 1997; Maddison et al., 2017).”
Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning (2017, background): “Within reinforcement learning, Bayesian notions of model uncertainty have a long history (Schmidhuber, 1991b;a; Dearden et al., 1999; Still & Precup, 2012; Sun et al., 2011; Maddison et al., 2017).”
Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables (2017, methodology): “(10) In practice, s0 is sampled uniformly at random from the training data D. Standard approaches in risk-sensitive RL (Garcı́a & Fernández, 2015; Maddison et al., 2017; Mihatsch & Neuneier, 2002) use the standard deviation of the cost C as risk measure.”
Decomposition of Uncertainty for Active Learning and Reliable Reinforcement Learning in Stochastic Systems (2017, background): “Within reinforcement learning, Bayesian notions of model uncertainty have a long history [Schmidhuber, 1991b,a, Dearden et al., 1999, Still and Precup, 2012, Sun et al., 2011, Maddison et al., 2017, Mihatsch and Neuneier, 2002].”