Risk-Sensitive Reinforcement Learning

By Yun Shen, Michael J. Tobia, Tobias Sommer, Klaus Obermayer. 2013

366 citations recorded by the source corpus.

We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received rewards but also to the true transition probabilities of the underlying Markov decision process. When appropriate utility functions are chosen, the agents’ behaviors express key features of human behavior as predicted by prospect theory (Kahneman & Tversky, 1979), for example, different risk preferences for gains and losses, as well as the shape of subjective probability curves. We derive a risk-sensitive Q-learning algorithm, which is necessary for modeling human behavior when transition probabilities are unknown, and prove its convergence. As a proof of principle for the applicability of the new framework, we apply it to quantify human behavior in a sequential investment task. We find that the risk-sensitive variant provides a significantly better fit to the behavioral data and that it leads to an interpretation of the subject's responses that is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals shows a significant correlation of the risk-sensitive TD error with BOLD signal change in the ventral striatum. In addition we find a significant correlation of the risk-sensitive Q-values with neural activity in the striatum, cingulate cortex, and insula that is not present if standard Q-values are used.

Subfields: Artificial Intelligence, Cognitive Neuroscience, General Decision Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In General Decision Sciences

Citations in the corpus

Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation (2018, background): “Nevertheless, these methods may recommend treatments that are obviously different from doctors’ prescriptions due to the lack of the supervision from doctors, which may cause high risk [22] in clinical practice.”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “(1) For τ ≤ 0 this utility function is risk-averse and has been studied extensively in the literature on safety and risk-averse reinforcement learning (Howard and Matheson, 1972; Pratt, 1975; Marcus et al., 1997; Mihatsch and Neuneier, 2002; Liu et al., 2003; Maddison et al., 2017).”
Safe Exploration in Markov Decision Processes (2012, neutral): “Copyright 2012 by the author(s)/owner(s).”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “This disconnect extends to deep RL, where the majority of state of the art algorithms have no concept of uncertainty [42, 41] and can fail spectacularly even in simple problems where success requires its consideration [40, 45].”
Safe Exploration of State and Action Spaces in Reinforcement Learning (2012, background): “As will be described in Section 5 in greater detail, this new deﬁnition is completely diﬀerent from traditional deﬁnitions of “risk” found in the literature (Geibel, 2001; Mihatsch & Neuneier, 2002; Geibel & Wysotzki, 2005).”
Recent advances in reinforcement learning in finance (2021, methodology): “…popular types of RL methods that have been used in optimal execution problems are 𝑄-learning algorithms and (double) DQN (Dabérius et al., 2019; Hendricks . . . Wilcox, 2014; Jeong & Kim, 2019; Ning et al., 2018; Cartea et al., 2021; Shen et al., 2014; Nevmyvaka et al., 2006; Zhang et al., 2020).”
Intelligent Resource Slicing for eMBB and URLLC Coexistence in 5G and Beyond: A Deep Reinforcement Learning Based Approach (2020, background): “We consider the exponential function that can capture both the mean and variance as defined in [26]:”
Dead-ends and Secure Exploration in Reinforcement Learning (2019, background): “…tries to assert an optimization criterion, such as worst case scenario (Heger, 1994), risk-sensitive criterion by transforming the TD error (Mihatsch & Neuneier, 2002), and call-time criterion by defining the safety in terms the probability of returning to the initial state (Moldovan &…”
A comprehensive survey on safe reinforcement learning (2015, background): “…of the approaches address finite MDPs (Heger, 1994a; Gaskett, 2003; Moldovan and Abbeel, 2012a) and problems with continuous or large state spaces (Mihatsch and Neuneier, 2002; Geibel and Wysotzki, 2005; Thomaz and Breazeal, 2008), and much fewer approaches address problems with continuous or…”
If MaxEnt RL is the Answer, What is the Question? (2019, methodology): “These methods then maximize the following likelihood: The last term is a cumulant generating function (i.e., the logarithm of a moment generating function [15, Chpt. 6]), which can be approximated as the sum of expected reward and variance of returns [30].”
Idiosyncrasies and challenges of data driven learning in electronic trading (2018, background): “The certainty equivalent ( CE ) modiﬁcation of the standard RL equation is (see also Bühler et al. [2018] and Mihatsch and Neuneier [2002]): where U and U − 1 is the utility function and its inverse, E denotes expectation, CE denotes certainty equivalent: is the policy π action in the state s i ,…”
Particle Value Functions (2017, background): “It has been extensively studied for the control of MDPs (Howard & Matheson, 1972; Coraluppi, 1997; Marcus et al., 1997; Borkar & Meyn, 2002; Mihatsch & Neuneier, 2002; B ¨ auerle & Rieder, 2013).”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, background): “However, unlike prior work (Heger, 1994; Shen et al., 2014; Tamar et al., 2014; 2015; Tang et al., 2019; Zhang et al., 2021), which focuses on risk with respect to a known reward function and stochastic transitions, we consider policy optimization when there is epistemic uncertainty over the reward…”
Ultra-Reliable Communication in 5G mmWave Networks: A Risk-Sensitive Approach (2018, methodology): “However, at very low rate (less than 2 Gbps) or very high rate (10.65 − 11 Gbps) captured by the cross-point, the RSL obtains a lower probability as compared to the baselines.”
Decomposition of Uncertainty for Active Learning and Reliable Reinforcement Learning in Stochastic Systems (2017, methodology): “…and Garcia and Fern´andez [2012] approaches the problem of safe exploration, the former by using a Lyapunov stability veriﬁcation and the latter by gradually improving over a predeﬁned (and safe) baseline policy (For a comprehensive review of risk-sensitive RL we refer to Shen et al. [2014]).”