Percentile Optimization for Markov Decision Processes with Parameter Uncertainty

By Erick Delage, Shie Mannor. 2010

271 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Statistics and Probability

Citations in the corpus

Safe Policy Improvement with Baseline Bootstrapping (2017, background): “We start from the percentile criterion (Delage & Mannor, 2010) on the safe policy improvement over the baseline π b : where P MDP ( ·”
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles (2016, methodology): “Risk sensitive RL methods (Delage & Mannor, 2010; Tamar et al., 2015) have been proposed to act as a bridge between robust control and Bayesian RL.”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “In particular, percentile-based measures have been utilised (Delage and Mannor, 2010; Filar et al., 1995) to ensure/target specific guarantees of a probability of obtaining a reward in a given task.”
Probabilistically safe policy transfer (2017, background): “Typical constraints involve ensuring that either the expectation of the return exceeds a threshold [9], [13], [23], the variance of the return remains below a threshold [4], or that any state is reachable from any other state (ergodicity) [15], [19].”
Risk Aversion in Markov Decision Processes via Near Optimal Chernoff Bounds (2012, background): “Percentile optimization has been shown to be tractable when dealing with ambiguity in MDP parameters [8, 9], and it has also been discussed in the context of risk [10, 11].”
A comprehensive survey on safe reinforcement learning (2015, background): “This chance-constraint metric, P (E(R) ≥ α) ≥ (1− ), is interpreted as guaranteeing that the expectation of the return (considered a random variable) will be at least as good as α with a probability greater than or equal to (1− ) (Delage and Mannor, 2010; Ponda et al., 2013).”
Bayesian Robust Optimization for Imitation Learning (2020, background): “We consider the machine replacement problem, a common problem in the robust MDP literature [14].”
Robust Adversarial Reinforcement Learning (2017, background): “Other approaches are risk-based including (Tamar et al., 2014; Delage & Mannor, 2010), which formulate various mechanisms of percentile risk into the formulation.”
A distributional interpretation of robust optimization (2010, background): “Xu and Mannor [42], Delage and Mannor [13]).”
Distributionally Robust Markov Decision Processes (2010, methodology): “Indeed, it was observed that since the robust MDP framework ignores probabilistic information, it can pr ovide conservative solutions [11, 12].”
Risk-Averse Bayes-Adaptive Reinforcement Learning (2021, background): “One approach is to find a policy which optimises the expected value in the worst k% of parameter realisations [5, 11].”
Bayesian Reinforcement Learning (2010, background): “Curtailing this problem, Delage and Mannor (2010) present an approximation to a risk-sensitive percentile optimization criterion: maximize y ∈ R , π ∈ ϒ y s . t .”
Interval Estimation for Reinforcement-Learning Algorithms in Continuous-State Domains (2010, background): “(2004) investigated confidence estimates for the parameters of the learned transition and reward models, assuming Gaussian rewards [5, 16].”
Convex Optimization: Algorithms and Complexity (2015, unknown): “[Delage and Mannor, 2010] Let N∗ = mins,a ∑ s′ n(s, a, s′) denote the minimum number of state transitions observed from any state using any action, and ∈ (0, 0.5].”
Parametric regret in uncertain Markov decision processes (2009, background): “[8]): The true parameters are regarded as random variables.”
Actor-Critic Algorithms for Risk-Sensitive MDPs (2013, background): “, [12, 7, 24]), and 2) the inherent uncertainty related to the stochastic nature of the system, which is the topic of risk-sensitive MDPs (e.”
Robust Markov Decision Processes (2013, background): “Let P 0 be the unknown true transition kernel of the MDP.”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, methodology): “Despite the popularity of VaR, optimizing a policy for VaR has several problems: (1) optimizing for VaR results in an NP hard optimization problem (Delage & Mannor, 2010), (2) VaR ignores risk in the tail that occurs with probability less than (1− α) which is problematic for domains where there are…”