Policy Gradients with Variance Related Risk Criteria

By Dotan Di Castro, Aviv Tamar, Shie Mannor. 2012

241 citations recorded by the source corpus.

Managing risk in dynamic decision problems is of cardinal importance in many fields such as finance and process control. The most common approach to defining risk is through various variance related criteria such as the Sharpe Ratio or the standard deviation adjusted reward. It is known that optimizing many of the variance related risk criteria is NP-hard. In this paper we devise a framework for local policy gradient style algorithms for reinforcement learning for variance related criteria. Our starting point is a new formula for the variance of the cost-to-go in episodic tasks. Using this formula we develop policy gradient algorithms for criteria that involve both the expected cost and the variance of the cost. We prove the convergence of these algorithms to local minima and demonstrate their applicability in a portfolio planning problem.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “The theorem extends the deterministic reward case [100, 108].”
Challenges of Real-World Reinforcement Learning (2019, background): “This has previously been applied in a number of different works (Mankowitz et al., 2018; Di Castro et al., 2012; Derman et al., 2018).”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “When thinking about general capability and catastrophic failure of policies, the field of robust and risk sensitive reinforcement learning (Borkar and Jain, 2010; Prashanth and Ghavamzadeh, 2013; Tamar et al., 2012) has been analysing variability in obtained rewards to find safer solutions.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “, [24, 36, 109, 116, 126, 173, 201, 240, 264]; some methods consider safety in RL exploration process by leveraging external knowledge, e.”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “Forexample,fromtheperspectiveofobjectiveoptimiza-tion,severalmethodsconsidercostasoneoftheoptimization objectivestoachievesafety,e.g.,[91],[92],[93],[94],[95],[96], [97],[98],[99].”
A comprehensive survey on safe reinforcement learning (2015, background): “…                                                                 Optimization Criterion                                                                         …”
Optimizing the CVaR via Sampling (2014, background): “Morimura et al. (2010) consider the expected return, with a CVaR based risk-sensitive policy for guiding the exploration while learning.”
Reward Constrained Policy Optimization (2018, background): “Variance [Tamar et al., 2012] and CVaR [Prashanth, 2014], such that convergence to a constraint satisfying policy is not ensured.”
Algorithms for CVaR Optimization in MDPs (2014, background): “[27] and Prashanth and Ghavamzadeh [17] on several variance-related risk measures.”
Probabilistically safe policy transfer (2017, background): “Typical constraints involve ensuring that either the expectation of the return exceeds a threshold [9], [13], [23], the variance of the return remains below a threshold [4], or that any state is reachable from any other state (ergodicity) [15], [19].”
Actor-Critic Algorithms for Risk-Sensitive MDPs (2013, background): “In comparison to [22], which is the closest related work, we would like to remark that while the authors there develop policy gradient methods for stochastic shortest path problems, we devise actorcritic algorithms for both discounted and average reward settings.”
Constrained Reinforcement Learning Has Zero Duality Gap (2019, methodology): “The combination of different rewards with manually selected Lagrange multipliers has been applied for instance to learning complex movements for humanoids [4] or to limit the variance of the constraint that needs to be satisfied [19, 20].”