Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach

By Yinlam Chow, Aviv Tamar, Shie Mannor, Marco Pavone. 2015

412 citations recorded by the source corpus.

In this paper we address the problem of decision making within a Markov decision process (MDP) framework where risk and modeling errors are taken into account. Our approach is to minimize a risk-sensitive conditional-value-at-risk (CVaR) objective, as opposed to a standard risk-neutral expectation. We refer to such problem as CVaR MDP. Our first contribution is to show that a CVaR objective, besides capturing risk sensitivity, has an alternative interpretation as expected cost under worst-case modeling errors, for a given error budget. This result, which is of independent interest, motivates CVaR MDPs as a unifying framework for risk-sensitive and robust decision making. Our second contribution is to present an approximate value-iteration algorithm for CVaR MDPs and analyze its convergence rate. To our knowledge, this is the first solution algorithm for CVaR MDPs that enjoys error guarantees. Finally, we present results from numerical experiments that corroborate our theoretical findings and show the practicality of our approach.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Distributional Reinforcement Learning with Quantile Regression (2017, neutral): “Recently, Bellemare, Dabney, and Munos (2017) showed that the distribution of the random returns, whose expectation constitutes the aforementioned value, can be described by the distributional analogue of Bellman’s equation, echoing previous results in risk-sensitive reinforcement learning (Heger 19”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “While there are abundant works on safety for RL (Chow et al., 2015; Eysenbach et al., 2018; Chow et al., 2018; Ray et al., 2019; Chow et al., 2019), they often do not provide guarantees in practice when combined with neural networks and stochastic optimization.”
Risk-Aware Active Inverse Reinforcement Learning (2018, background): “many different problems including planning [10, 11], reinforcement learning [12, 13], and inverse reinforcement learning [14].”
Scaling up budgeted reinforcement learning (2019, background): “…include other statistics of the performance, such as the worst-case realisation [Iyengar, 2005; Nilim and El Ghaoui, 2005; Wiesemann et al. , 2013], the variance-penalised expectation [Garc´ıa and Fern´andez, 2015; Tamar et al. , 2012], or the Conditional Value-At-Risk (CVaR) [Chow et al. , 2015b].”
Robust artificial intelligence and robust human organizations (2018, methodology): “They should incorporate techniques, such as minimizing down-side risk, that confer robustness to model error [4].”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, background): “Recently, objectives based on measures of risk such as value at risk (VaR) and conditional VaR have been shown to provide tractable and useful risk-sensitive measures of performance for MDPs (Tamar et al., 2015; Chow et al., 2015).”
Bayesian Robust Optimization for Imitation Learning (2020, background): “This work has been extended to risk-averse optimization in Markov decision processes [13, 21, 41] and in the context of reinforcement learning [20, 55, 56], where the transition dynamics and reward function are not known.”
Risk-sensitive Inverse Reinforcement Learning via Coherent Risk Models (2017, methodology): “This theory has also recently been adopted for risk-sensitive (RS) Model Predictive Control and decision making [12, 13], and autonomous exploration [8].”
Measuring and avoiding side effects using relative reachability (2018, background): “…by enforcing safety constraints (Turchetta et al., 2016; Dalal et al., 2018), using intrinsic motivation (Lipton et al., 2016), penalizing risk (Chow et al., 2015; Mihatsch and Neuneier, 2002), preserving ergodicity (Moldovan and Abbeel, 2012; Eysenbach et al., 2017), etc. Explicitly defined…”
Recent advances in reinforcement learning in finance (2021, background): “Another line of work focuses on constrained RL problems with different risk criteria [4, 48, 49, 64, 219, 267].”
Constrained Reinforcement Learning Has Zero Duality Gap (2019, background): “CMDP applications cover a vast number of topics, such as: electric grids [14], networking [15], robotics [3, 16, 17] and finance [18, 19].”
Combating Deep Reinforcement Learning ’ s Sisyphean Curse with Reinforcement Learning (2017, methodology): “Applications of this appraoch to RL in clude Mannor and Tsitsiklis [2011] and Chow et al. [2015]. The second category describes papers which incorporate external knowledge into the exploration process.”
Reward Constrained Policy Optimization (2018, methodology): “CMDP applications cover a vast number of topics, such as: electric grids (Koutsopoulos and Tassiulas, 2011), networking (Hou and Zhao, 2017), robotics (Chow et al., 2015; Gu et al., 2017; Achiam et al., 2017; Dalal et al., 2018) and ﬁnance (Krokhmal et al., 2002; Tamar et al., 2012).”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, neutral): “Although the empirical results on MuJoCo [3] and grid-world environments [144] have demonstrated their method’s effectiveness, the theoretical analysis to guarantee the performance is still needed to be provided.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “Although the empirical results on MuJoCo [4] and grid-world environments [58] have demonstrated the effectiveness of their method, the theoretical analysis to guarantee the performance is still needed to be provided.”
Risk versus Uncertainty in Deep Learning: Bayes, Bootstrap and the Dangers of Dropout (2016, background): “For exploitation, any concave utility will naturally penalize risk [4].”
Risk-Averse Bayes-Adaptive Reinforcement Learning (2021, methodology): “To address the continuous state variable, y, [8] use dynamic programming with linear function approximation.”
Penalizing Side Effects using Stepwise Relative Reachability (2018, background): “, 2016], penalizing risk [Chow et al., 2015], using intrinsic motivation [Lipton et al.”
Avoiding Side Effects By Considering Future Tasks (2020, background): “Safe exploration methods that penalize risk [4] or use intrinsic motivation [14] help the agent avoid side effects that result in lower reward (such as getting trapped or damaged), but do not discourage the agent from damaging the environment in ways that are not penalized by the reward function (e.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “The principle axis of variation regarding performance metrics is whether the expected return of a policy is being bounded, or whether a risk-aware function of return is used—for example, a robust worst-case analysis [122], a PAC-style bound [360], or bounded value-at-risk [66, 44].”