Optimizing the CVaR via Sampling

By Aviv Tamar, Yonatan Glassner, Shie Mannor. 2014

229 citations recorded by the source corpus.

Conditional Value at Risk (CVaR) is a prominent risk measure that is being used extensively in various domains. We develop a new formula for the gradient of the CVaR in the form of a conditional expectation. Based on this formula, we propose a novel sampling-based estimator for the gradient of the CVaR, in the spirit of the likelihood-ratio method. We analyze the bias of the estimator, and prove the convergence of a corresponding stochastic gradient descent algorithm to a local CVaR optimum. Our method allows to consider CVaR optimization in new domains. As an example, we consider a reinforcement learning application, and learn a risk-sensitive controller for the game of Tetris.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Wireless Network Intelligence at the Edge (2018, background): “, CVAR, by observing extra samples as done in [147].”
Challenges of Real-World Reinforcement Learning (2019, methodology): “One evaluation procedure is to determine whether rare catastrophic rewards are minimized (Tamar et al., 2015b;a).”
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles (2016, methodology): “For the CVaR case, Tamar et al. (2015) showed that without using a baseline, the policy gradient is biased.”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “Other work focuses on ﬁnding robust solutions to MDPs (Ghavamzadeh et al., 2016; Petrik & Russell, 2019), us-ing model-based RL to safely improve upon suboptimal demonstrations (Thananjeyan et al., 2019), and obtaining high-conﬁdence off-policy bounds on the performance of an evaluation policy (Thom”
Reinforcement Learning under Model Mismatch (2017, background): “Other probabilistic and risk based settings were studied by [11, 28, 25] which propose various mechanisms to incorporate percentile risk into the model.”
Risk-Averse Bayes-Adaptive Reinforcement Learning (2021, methodology): “Other works propose policy gradient methods to find a locally optimal solution for the optimisation of CVaR [4, 6, 7, 31, 43, 45, 46], or general coherent risk metrics [44].”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “…et al. [231] deploy safe RL to optimize the healthy recommendation sequences of recommender systems by utilizing a policy gradient method algorithm on the Conditional Value at Risk (CVaR) method [236], whereby they optimize positive feedback while constraining cumulative exposure of health risk.”
Measuring the Reliability of Reinforcement Learning Algorithms (2019, methodology): “Originally developed in ﬁnance, CVaR has also seen recent adoption in Safe RL as an additional component of the objective function by applying it to the cumulative returns within an episode, e.g. Bäuerle & Ott (2011); Chow & Ghavamzadeh (2014); Tamar et al. (2015).”
Bayesian Robust Optimization for Imitation Learning (2020, background): “This work has been extended to risk-averse optimization in Markov decision processes [13, 21, 41] and in the context of reinforcement learning [20, 55, 56], where the transition dynamics and reward function are not known.”
Risk-Aware Active Inverse Reinforcement Learning (2018, background): “many different problems including planning [10, 11], reinforcement learning [12, 13], and inverse reinforcement learning [14].”
Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients (2019, methodology): “…and the deﬁnition of conditional expectation yields the ﬁnal result: The extension to the case in which Z is replaced by a scalar reward function R ( τ ) , where τ is generated from a parameterized distribution p ( τ | θ ) , follows the proof in Tamar et al. (2014) without additional adaptation.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “[254] deploy safe RL to optimise the healthy recommendation sequences of recommender systems by utilising a policy gradient method algorithm on the Conditional Value at Risk (CVaR) method [266], whereby they optimise positive feedback whilst constraining cumulative exposure of health risk.”
Unicorn: Continual Learning with a Universal, Off-policy Agent (2018, background): “Additions include optimizing for risk-aware, [46], robust [45, 27] as well as exploratory [6, 31] objectives to yield improved performance.”
Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach (2015, background): “In contrast to the algorithms in [4, 8, 5, 25], given the explicit MDP model our approach leads to finite-time error guarantees, with respect to the globally optimal policy.”
Assessing Generalization in Deep Reinforcement Learning (2018, background): “A popular approach to learn a robust policy is to maximize a risk-sensitive objective, such as the conditional value at risk (Tamar et al., 2015), over a distribution of environments.”
AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, background): “For the ﬁrst line, the focus is on learning policies that are robust to environment variations, e.g., by maximizing a risk-sensitive objective over a distribution of environments (Tamar et al., 2015) or by extracting a set of invariant states (Zhang et al., 2020a; 2021a; Tomar et al., 2021).”
On Ensuring that Intelligent Machines Are Well-Behaved (2017, unknown): “Alternatively, risk-sensitive methods might define f(θ) to be the expected shortfall or conditional value at risk (CVaR) of the solution θ—a measure of the expected performance of the robot during the worst trials, when using the controller θ [Tamar et al., 2014; Chow and Ghavamzadeh, 2014].”
Robust Deep Reinforcement Learning with Adversarial Attacks (2017, unknown): “Thus, conditional value of risk (CVaR) can be used as optimization criteria for robust control (Tamar et al. [2015]) So, the problem boils down to maximizing the expected return over worst α percentile of returns.”