Risk-Constrained Reinforcement Learning with Percentile Risk Criteria

By Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, Marco Pavone. 2015

660 citations recorded by the source corpus.

In many sequential decision-making problems one is interested in minimizing an expected cumulative cost while taking into account \emph{risk}, i.e., increased awareness of events of small probability and high consequences. Accordingly, the objective of this paper is to present efficient reinforcement learning algorithms for risk-constrained Markov decision processes (MDPs), where risk is represented via a chance constraint or a constraint on the conditional value-at-risk (CVaR) of the cumulative cost. We collectively refer to such problems as percentile risk-constrained MDPs. Specifically, we first derive a formula for computing the gradient of the Lagrangian function for percentile risk-constrained MDPs. Then, we devise policy gradient and actor-critic algorithms that (1) estimate such gradient, (2) update the policy in the descent direction, and (3) update the Lagrange multiplier in the ascent direction. For these algorithms we prove convergence to locally optimal policies. Finally, we demonstrate the effectiveness of our algorithms in an optimal stopping problem and an online marketing application.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Constrained Policy Optimization (2017, methodology): “Our method has similar policy updates to primal-dual methods like those proposed by Chow et al. (2015), but crucially, we differ in computing the dual variables (the Lagrange multipliers for the constraints).”
Recent advances in reinforcement learning in finance (2021, background): “Another line of work focuses on constrained RL problems with different risk criteria [4, 48, 49, 64, 215, 263].”
Lyapunov-based Safe Policy Optimization for Continuous Control (2019, methodology): “A common approach to solve CMDPs is to use the Lagrangian method (Altman, 1998; Geibel & Wysotzki, 2005) that augments the original objective function with a penalty on constraint violation and computes the saddlepoint of the constrained policy optimization via primal-dual methods (Chow et al., 2017”
Scaling up budgeted reinforcement learning (2019, background): “on [Iyengar, 2005; Nilim and El Ghaoui, 2005; Wiesemann et al., 2013], the variance-penalised expectation [Garc´ıa and Fern ´andez, 2015; Tamar et al., 2012], or the Conditional Value-At-Risk (CVaR) [Chow et al., 2015b]. Conﬂicting objectives. Reinforcement Learning also assumes that the performance”
A Tutorial on Ultrareliable and Low-Latency Communications in 6G: Integrating Domain Knowledge Into Deep Learning (2021, background): “A long-term cost could be the average queuing delay [272], [273], the access cost for handovers [274], the visitation probability of some states [275], or the cost in the tail of a risk distribution [276].”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, positive): “The following optimization objective is a Safe RL scheme previously outlined in Chow et al. (2017), hereby defined as the objective for our Safe RLHF setting: where D is a distribution of prompts used in the RL phase, and the y = a 1: T are responses generated by the LLM π θ .”
Risk-Averse Bayes-Adaptive Reinforcement Learning (2021, methodology): “Other works propose policy gradient methods to find a locally optimal solution for the optimisation of CVaR (Borkar & Jain, 2010; Chow & Ghavamzadeh, 2014; Tamar et al., 2015b;a; Prashanth, 2014; Chow et al., 2017; Tang et al., 2019), or general coherent risk metrics (Tamar et al.”
Uncertainty Estimation for Language Reward Models (2022, background): “Estimating reward model uncertainty allow us to instead perform risk-averse [45, 42] or risk-constrained [8] reinforcement learning.”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “In particular, (88) defines a constraint on the CVaR of cost J, and uses policy gradient or actor-critic methods to update the policy in Equation 15b.”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, positive): “In addition, based on our investigation, in MARL settings, [174] is one of the earliest methods that present convergence guarantee for multi-agent systems in year 2000.”
A Lyapunov-based Approach to Safe Reinforcement Learning (2018, methodology): “In this paper, we follow the standard notion of transient MDPs and assume that the first-hitting time is uniformly bounded by an upper bound T for any stationary policies [10].”
Projection-Based Constrained Policy Optimization (2020, background): “(2) Primal-dual Optimization (PDO) (Chow et al., 2017).”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “[57] to optimize risk RL performance, in which CVaR4-constrained and chance-constrained optimization are used to guarantee safety.”