Algorithms for CVaR Optimization in MDPs

By Yinlam Chow, Mohammad Ghavamzadeh. 2014

236 citations recorded by the source corpus.

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively new risk measure that addresses some of the shortcomings of the well-known variance-related risk measures, and because of its computational efficiencies has gained popularity in finance and operations research. In this paper, we consider the mean-CVaR optimization problem in MDPs. We first derive a formula for computing the gradient of this risk-sensitive objective function. We then devise policy gradient and actor-critic algorithms that each uses a specific method to estimate this gradient and updates the policy parameters in the descent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in an optimal stopping problem.

Subfields: Management Science and Operations Research, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Ocean Engineering

Citations in the corpus

Implicit Quantile Networks for Distributional Reinforcement Learning (2018, methodology): “Finally, we consider conditional value-at-risk (CVaR): CVaR has been widely studied in and out of reinforcement learning (Chow & Ghavamzadeh, 2014).”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, background): “CVnaR, Conditional Value-not-at-Risk, is inspired by well studied risk-averse operator Conditional Value-at-Risk (CVaR ( η, τ ) = ητ ) (Chow & Ghavamzadeh, 2014).”
Risk-Averse Bayes-Adaptive Reinforcement Learning (2021, background): “Other works propose policy gradient methods to find a locally optimal solution for the optimisation of CVaR [4, 6, 7, 31, 43, 45, 46], or general coherent risk metrics [44].”
Reward Constrained Policy Optimization (2018, background): “While the assumption above might seem unnatural, it is not uncommon [Chow and Ghavamzadeh, 2014]; we note that in practice, most reinforcement learning algorithms gather sample trajectories with bounded stopping time (e.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, methodology): “use a conditional value-at-risk (CVaR [87]) objective to learn a risk-averse policy.”
Preventing undesirable behavior of intelligent machines (2019, background): “Alternatively, risk-sensitive methods might define f(θ) to be the expected shortfall or conditional value at risk (CVaR) of the solution θ, a measure of the expected performance of the robot during the worst trials when using controller θ [86, 87].”
Measuring the Reliability of Reinforcement Learning Algorithms (2019, methodology): “Originally developed in ﬁnance, CVaR has also seen recent adoption in Safe RL as an additional component of the objective function by applying it to the cumulative returns within an episode, e.g. Bäuerle & Ott (2011); Chow & Ghavamzadeh (2014); Tamar et al. (2015).”
On Ensuring that Intelligent Machines Are Well-Behaved (2017, unknown): “Alternatively, risk-sensitive methods might define f(θ) to be the expected shortfall or conditional value at risk (CVaR) of the solution θ—a measure of the expected performance of the robot during the worst trials, when using the controller θ [Tamar et al., 2014; Chow and Ghavamzadeh, 2014].”
QUOTA: The Quantile Option Architecture for Reinforcement Learning (2018, background): “…Classic risk-sensitive RL is also based on the intrinsic uncertainty, where a utility function (Morgenstern and Von Neumann 1953) is used to distort a value distribution, resulting in risk-averse or risk-seeking policies (Howard and Math-eson 1972; Marcus et al. 1997; Chow and Ghavamzadeh 2014).”
Risk-Constrained Reinforcement Learning with Percentile Risk Criteria (2015, background): “This paper generalizes earlier results by the authors presented in Chow and Ghavamzadeh (2014).”
RAIL: Risk-Averse Imitation Learning (2017, methodology): “Chow and Ghavamzadeh (2014) developed policy gradient and actor-critic algorithms for mean-CV aR optimization for learning policies in the classic RL setting.”
Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning (2018, background): “CVnaR, Conditional Value-not-at-Risk, is inspired by well studied risk-averse operator Conditional Value-at-Risk (CVaR ( 𝜂,𝜏 ) = 𝜂𝜏 ) [4].”
The effects of negative adaptation in Model-Agnostic Meta-Learning (2018, background): “Working with this kind of constrained objective relates to safety issues in reinforcement learning [2].”
Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach (2015, background): “In contrast to the algorithms in [4, 8, 5, 25], given the explicit MDP model our approach leads to finite-time error guarantees, with respect to the globally optimal policy.”