Robustness in Markov Decision Problems with Uncertain Transition Matrices

By Arnab Nilim, Laurent El Ghaoui. 2003

141 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “Robust MDPs [24, 50, 56, 44] have been a popular theoretical abstraction for offline RL, but tend to be highly conservative in policy improvement.”
Reinforcement Learning with Limited Reinforcement : Using Bayes Risk for Active Learning in POMDPs Finale (2012, background): “10A somewhat related area of work is planning with imprecise pa rameters [42, 43], but unlike learning approaches, this area assumes that additional int eractions with the environment cannot reduce model-uncertainty.”
Approximate MaxEnt Inverse Optimal Control and Its Application for Mental Simulation of Human Interactions (2015, methodology): “Our formulation focuses on the prediction of decision, but similar model can also arise from information-theoretical constraints on decision making (Nilim and Ghaoui 2003; Todorov 2006; Theodorou and Todorov 2012).”
Robust Dynamic Programming (2005, result): “The results in this section, although independently obtained, are not new and were first obtained by Nilim and El Ghaoui [17] (see Nilim and El Ghaoui [18]).”
An analysis of model-based Interval Estimation for Markov Decision Processes (2008, background): “The problem of finding a policy that is no worse than the optimal policy for any MDP whose transition function falls in the given confidence interval is interesting but different than our problem and has been well studied (Givan et al., 2000; Nilim and Ghaoui, 2004).”
Reinforcement Learning under Model Mismatch (2017, methodology): “In this setting, [17, 13] prove the following robust analogue of the Bellman recursion .”
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, background): “Compared to our SA-MDP where the adversary changes only observations, in RMDP the ground-truth states are changed so RMDP is more suitable for modeling environment parameter changes (e.g., changes in physical parameters like mass and length, etc).”
Robust Reinforcement Learning (2005, unknown): “A similar approach to the methods using dynamics programming was proposed by Nilim and Ghaoui (2004) to cope with uncertain state transition matrices.”
Assessing Generalization in Deep Reinforcement Learning (2018, background): “Nilim & Ghaoui (2004) assume that the set of possible transition probability distribution functions are known, while Lim et al. (2013) and Roy et al. (2017) estimate it using sampled trajectories from the distribution of environments of interest.”
Distributionally Robust Markov Decision Processes (2010, background): “Many efforts have been made to reduce such performance variation under the robust MDP framework (e.g., [5] [6] [7] [8] [9] [10]).”