Robust Dynamic Programming

By Garud N. Iyengar. 2005

901 citations recorded by the source corpus.

Subfields: Economics and Econometrics, Management Science and Operations Research, Ocean Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Management Science and Operations Research

In Ocean Engineering

In Reinforcement Learning

Citations in the corpus

Concrete Problems in AI Safety (2016, background): “ are being considered. For this, one could potentially use a conservative but reliable baseline policy, similar to the robust policy improvement and reachability analysis approaches discussed earlier [91, 98, 71, 109]. The ideal outcome of these approaches to limiting side-eﬀects would be to prevent”
Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Robust MDPs [24, 50, 56, 44] have been a popular theoretical abstraction for offline RL, but tend to be highly conservative in policy improvement.”
Safe Policy Improvement with Soft Baseline Bootstrapping (2019, background): “…RL competitors: basic RL, High Confidence Policy Improvement (Thomas, 2015, HCPI), Reward-Adjusted MDPs (Petrik et al., 2016, RaMDP), Robust MDPs (Iyengar, 2005; Nilim and El Ghaoui, 2005), and to Soft-SPIBB natural parents: Πb-SPIBB and Π≤b-SPIBB (Laroche et al., 2019); and on a helicopter…”
Safe Policy Improvement with Baseline Bootstrapping (2017, critical): “In place of the intractable expectation in Equation (1), Robust MDP classically consider optimizing the policy performance ρ ( π, M ) of the worst-case scenario in Ξ : In our benchmarks, we use the Robust MDP solver described in Petrik et al. (2016).”
Safe Policy Improvement by Minimizing Robust Baseline Regret (2016, methodology): “The main difference is the availability of the baseline policy that creates unique challenges for sequential optimization.”
Safe Policy Improvement with an Estimated Baseline Policy (2019, methodology): “the consequences would be disastrous as it would jeopardize a high number of systems, or even lives. Several attempts have been made to design reliable batch RL algorithms, starting with robust MDPs (Iyengar 2005; Nilim and El Ghaoui 2005), which consist of considering the set of plausible MDPs give”
Challenges of Real-World Reinforcement Learning (2019, background): “A Robust MDP is deﬁned by a tuple hS , A , P , r, γ i where S, A, r and γ are as previously deﬁned; P is a set of transition matrices referred to as the uncertainty set (Iyengar, 2005).”
A comprehensive survey on safe reinforcement learning (2015, background): “Specifically, a robust MDP deals with uncertainties in parameters; that is, some of the parameters, namely, transition probabilities, of the MDP are not known exactly (Bagnell et al., 2001; Iyengar, 2004; Nilim and El Ghaoui, 2005).”
RAAM: The Benefits of Robustness in Approximating Aggregated MDPs in Reinforcement Learning (2014, methodology): “RMDPs extend MDPs to allow uncertain transition probabilities and rewards and preserve most of the favorable MDP properties (Iyengar, 2005; Nilim and Ghaoui, 2005; Le Tallec, 2007; Wiesemann et al. , 2013).”
Scaling Up Robust MDPs by Reinforcement Learning (2013, methodology): “Formally, we deﬁne the robust value function [10, 14] for a policy π as its worst-case value function and we seek for the optimal robust value function V ∗ ( x ) = sup π inf P ∈P V π,P ( x ) .”
Bayesian Reinforcement Learning (2010, methodology): “This can be done ef-ﬁciently using dynamic programming style algorithms; see Nilim and El Ghaoui (2005); Iyengar (2005).”
Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach (2015, background): “In that study, the iterated (dynamic) coherent risk was shown to be equivalent to a robust MDP [10] with a rectangular uncertainty set.”
Action Robust Reinforcement Learning and Applications in Continuous Control (2019, methodology): “Furthermore, and to the best of our knowledge, in this line of works, discussion on the existence of strong-duality does not exists (i.e., as Proposition 1 and 4 assert for PR- and NR-MDPs).”
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (2020, methodology): “Minimax training has been used in the controls literature [37, 3, 49], the Robust MDP literature [4, 16, 26, 39], and”
Parametric regret in uncertain Markov decision processes (2009, background): “of the modeling parameters from the unknown true ones (cf [4]‐[ 7 ]).”
Learning Robust Options (2018, background): “We focus on (3): Learning policies in dynamical systems , using the robust MDP framework (Bagnell, Ng, and Schneider 2001; Nilim and El Ghaoui 2005; Iyenger 2005), that are robust to model uncertainty (e.g., robots with different arm lengths).”
Retailer-Supplier Flexible Commitments Contracts: A Robust Optimization Approach (2005, background): “For example, if the retailer only commits to a minimal cumulative quantity L for the entire horizon (as in Bassok and Anupindi 1997) we would set”
Applications of reinforcement learning in energy systems (2021, background): “There are works that build on the robust MDP framework [171], for example [172], and [173], whereas some other leverages on the equivalence between action-robust and robust MDPs introduced by Ref.”
Scaling Up Robust MDPs using Function Approximation (2014, methodology): “Thus, in the sequel we shall only consider deterministic policies, and write π(x) as the action prescribed by policy π at state x. Iyengar (2005) proposed a policy iteration algorithm for the robust MDP framework.”