Consideration of Risk in Reinforcement Learning

By Matthias Heger. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Distributional Reinforcement Learning with Quantile Regression (2017, neutral): “Recently, Bellemare, Dabney, and Munos (2017) showed that the distribution of the random returns, whose expectation constitutes the aforementioned value, can be described by the distributional analogue of Bellman’s equation, echoing previous results in risk-sensitive reinforcement learning (Heger 19”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, unknown): “, [24, 36, 109, 116, 126, 173, 201, 240, 264]; some methods consider safety in RL exploration process by leveraging external knowledge, e.”
MDPs: Learning in Varying Environments (2002, background): “…MDP model is that a wide range of models, e.g., Markov games (Littman, 1994), alternating Markov games (Boyan, 1992), discounted expected-reward MDPs (Watkins and Dayan, 1992), risk-sensitive MDPs (Heger, 1994), exploration-sensitive MDPs (John, 1994) can be discussed in this unified framework.”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, background): “However, unlike prior work (Heger, 1994; Shen et al., 2014; Tamar et al., 2014; 2015; Tang et al., 2019; Zhang et al., 2021), which focuses on risk with respect to a known reward function and stochastic transitions, we consider policy optimization when there is epistemic uncertainty over the reward ”
Safe Exploration Algorithms for Reinforcement Learning Controllers (2018, methodology): “This worst case or minimax [18] approach belongs to the optimization criterion of safety [19].”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “Forexample,fromtheperspectiveofobjectiveoptimiza-tion,severalmethodsconsidercostasoneoftheoptimization objectivestoachievesafety,e.g.,[91],[92],[93],[94],[95],[96], [97],[98],[99].”
Multi-criteria Reinforcement Learning (1998, methodology): “The analogous of Q-learning for MDPs with the maximin criterion, pro-posed by Heger (Heger, 1994, 1996), is the Q-hat algorithm de ned asQt+1(xt; at) = min Qt(xt; at); Rt(xt; at; xt+1) + maxb2A Qt(xt+1; b) :This algorithm will converge to the optimal Q-function ifQ0 Q (the initialestimate is optimis”
Truncating Temporal Diierences: on the Eecient Implementation of Td for Reinforcement Learning (1995, background): “Other reinforcement learning performance measures have also been considered Heger, 1994; Schwartz, 1993; Singh, 1994, but in this work we limit ourselves exclusively to the performance measure speciied by Equation 1.”
Risk-Sensitive Reinforcement Learning (2013, background): “Risk-sensitive decision-making problems, in the context of MDPs, have been investigated in various ﬁelds, e.g., in machine learning (Heger, 1994; Mihatsch and Neuneier, 2002), optimal control (Hern´andez-Hern´andez and Marcus, 1996), operations research (Howard and Matheson, 1972; Borkar, 2002),…”
Safe exploration for reinforcement learning (2008, unknown): “On ﬁrst sight, works as [3, 4, 5] seem to be similar to ours.”
A Unified Analysis of Value-Function-Based Reinforcement-Learning Algorithms (1999, methodology): “In this section, we have proven Theorem 11 concerning the convergence of Q-learning under a worst-case total discounted cost criterion, first stated by Heger (1994).”
Combating Deep Reinforcement Learning ’ s Sisyphean Curse with Reinforcement Learning (2017, background): “Along these lines, Heger [1994] suggested an alternative Q̂-learning objective, as follows:”
A Generalized Reinforcement-Learning Model: Convergence and Applications (1996, methodology): “.3 RISK-SENSITIVE MODELS Heger (1994) described an optimality criterion for mdps in which only the worst possible value of the next state makes a contribution to the value of a state.”
Algorithms for Sequential Decision Making (1996, background): “The operators in the risk-sensitive mdp modeldescribed by Heger [62] are polynomially expressible.”
Human Insula Activation Reflects Risk Prediction Errors As Well As Risk (2008, background): “Examples include nonlinear transformation of rewards [the expected utility approach (Koenig and Simmons, 1994)], max/min policies [robust control (Heger, 1994)], and sign-based weighting of prediction errors [effectively inducing loss aversion (Mihatsch and Neuneier, 2002)].”
Generalized Markov Decision Processes: Dynamic-programming and Reinforcement-learning Algorithms (1996, methodology): “Generalized mdps provide a foundation for decision making in mdps and games, as well as in risk-sensitive models [15], exploration-sensitive models [20, 36], simultaneous-action games [39], and other models.”