Risk-Sensitive Reinforcement Learning Applied to Control under Constraints

By Peter Geibel, Fritz Wysotzki. 2005

331 citations recorded by the source corpus.

In this paper, we consider Markov Decision Processes (MDPs) with error states. Error states are those states entering which is undesirable or dangerous. We define the risk with respect to a policy as the probability of entering such a state when the policy is pursued. We consider the problem of finding good policies whose risk is smaller than some user-specified threshold, and formalize it as a constrained MDP with two criteria. The first criterion corresponds to the value function originally given. We will show that the risk can be formulated as a second criterion function based on a cumulative return, whose definition is independent of the original value function. We present a model free, heuristic reinforcement learning algorithm that aims at finding good deterministic policies. It is based on weighting the original value function and the risk. The weight parameter is adapted in order to find a feasible solution for the constrained problem that has a good performance with respect to the value function. The algorithm was successfully applied to the control of a feed tank with stochastic inflows that lies upstream of a distillation column. This control task was originally formulated as an optimal control problem with chance constraints, and it was solved under certain assumptions on the model to obtain an optimal solution. The power of our learning algorithm is that it can be used even when some of these restrictive assumptions are relaxed.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Safe Policy Improvement with Baseline Bootstrapping (2017, background): “In short, internal safety guarantees a certain level of return for each individual trajectory, which is critical for potential harmful behaviour or catastrophe avoidance scenarios [26, 27, 1, 8, 17].”
Safe Model-based Reinforcement Learning with Stability Guarantees (2017, background): “For example, [8] define risk as the probability of driving the agent to a set of known, undesirable states.”
A comprehensive survey on safe reinforcement learning (2015, methodology): “, 2002), or as the probability of entering into an error state (Geibel and Wysotzki, 2005).”
Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (2020, methodology): “The Lagrangian multiplier method is readily adapted to the constrained RL setting (Altman, 1998; Geibel & Wysotzki, 2011) and has become a popular baseline in deep RL (Achiam et al., 2017; Chow et al., 2019) for its simplicity and effectiveness.”
Safe Exploration in Finite Markov Decision Processes with Gaussian Processes (2016, background): “For example, Geibel and Wysotzki [7] define risk as the probability of driving the system to a previously known set of undesirable states.”
A Survey of Multi-Objective Sequential Decision-Making (2013, methodology): “Hence, multi-objective probabilistic planning has strong parallels with the research into risk-aware RL carried out by Geibel (2001) and Geibel and Wysotzki (2005), which add a second reward signal indicating the transition of the environment into an error state.”
Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones (2020, methodology): “…algorithms for safe RL involves modifying the policy optimization procedure of standard RL algorithms to simultaneously reason about both task reward and constraints using methods such as trust regions [1], optimizing a Lagrangian relaxation [25, 30, 34], or constructing Lyapunov functions [6, 7].”
Safe Exploration of State and Action Spaces in Reinforcement Learning (2012, methodology): “…PI-SRL in the four domains are compared to those yielded by two additional techniques, namely, the evolutionary RL approach selected winner of the helicopter domain in the 2009 RL Competition (Mart´ın H. & Lope, 2009) and Geibel and Wysotzki’s risk-sensitive RL approach (Geibel & Wysotzki, 2005).”
Safe Exploration Algorithms for Reinforcement Learning Controllers (2018, background): “Alterna-tively, by assigning negative reward, the variance of the return can be taken into account by adopting risk-sensitivity approaches [21]–[24].”
Learning the Variance of the Reward-To-Go (2016, background): “The variance of the reward-to-go may thus prove valuable for risk aware optimization algorithms, a topic that has gained significant interest recently (Filar et al., 1995; Mihatsch and Neuneier, 2002; Geibel and Wysotzki, 2005; Mannor and Tsitsiklis, 2013).”
A practical guide to multi-objective reinforcement learning and planning (2021, methodology): “In [Geibel and Wysotzki, 2005], [Geibel, 2006] and [Horie et al.”
Neuroevolutionary reinforcement learning for generalized control of simulated helicopters (2011, methodology): “Existing methods for risk-sensitive reinforcement learning [24, 53], are not applicable to pol-”
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, background): “Similar state-avoidant approaches to safety have been proposed by several others [89, 90, 91].”
Nonparametric Return Distribution Approximation for Reinforcement Learning (2010, background): “…the expected return may accept rare occurrences of large negative outcomes, some users, especially those who plan to apply a RL algorithm to practical applications, prefer a policy avoiding (or constraining) small chances of suffering a large loss (Geibel & Wysotzki, 2005; Defourny et al., 2008).”
Reinforcement Learning: A Tutorial Survey and Recent Advances (2009, unknown): “Other areas that are attracting interest are: risk-sensitive RL (Borkar, 2002; Geibel and Wysotzki, 2005), factored MDPs (Schuurmans and Patrascu, 2002), and analyzing computational complexity (Evan-Dar and Mansour, 2003 For a regular Markov chain, there exists a finite value for k, where k > 0 and…”
Lyapunov-based Safe Policy Optimization for Continuous Control (2019, methodology): “A common approach to solve CMDPs is to use the Lagrangian method (Altman, 1998; Geibel & Wysotzki, 2005) that augments the original objective function with a penalty on constraint violation and computes the saddle-point of the constrained policy optimization via primal-dual methods (Chow et al.,…”
A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems (2017, background): “In risk-sensitive reinforcement learning [12], the expected return was heuristically weighted with the probability (risk) of reaching an “error state”; while this allowed for more gen-”