Robust Control of Markov Decision Processes with Uncertain Transition Matrices

By Arnab Nilim, Laurent El Ghaoui. 2005

972 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Transportation, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Transportation

In Reinforcement Learning

Citations in the corpus

Concrete Problems in AI Safety (2016, background): “This idea is typically not taken literally, but it really is true that the reward, even when it is an abstract idea like the score in a board game, must be computed somewhere, such as a sensor or a set of transistors.”
Safe Policy Improvement with Baseline Bootstrapping (2017, methodology): “(Petrik et al., 2016) use Robust MDP (Iyengar, 2005; Nilim & El Ghaoui, 2005) to bound from below the constraint in (1) by considering a set of admissible MDPs Ξ = Ξ( (cid:99) M, e ) deﬁned as: where e : X × A → R is an error function depending on D and δ .”
Safe Policy Improvement by Minimizing Robust Baseline Regret (2016, methodology): “To model the uncertainty in the transition probability, we adopt the notion of robust MDP (RMDP) [5, 8, 15], i.e., an extension of MDP in which nature adversarially chooses the transitions from a given uncertainty set From Assumption 1, we notice that the true transition probability is in the set…”
Safe Exploration in Markov Decision Processes (2012, positive): “In (Nilim & El Ghaoui, 2005) they propose a robust MDP control method assuming the transition frequencies are drawn from an orthogonal convex set by an adversary.”
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles (2016, background): “Robust control is a branch of control theory which formally studies development of robust policies [15, 16].”
High Confidence Off-Policy Evaluation with Models (2016, neutral): “Model bias in MDPs can come from two sources: 1.”
Quantifying Distributional Model Risk via Optimal Transport (2016, neutral): “…Delage and Ye (2010); Goh and Sim (2010); Wiesemann et al. (2014) for moment-based uncertainty sets, Hansen and Sargent (2001); Iyengar (2005); Nilim and El Ghaoui (2005); Lim and Shanthikumar (2007); Jain et al. (2010); Ben-Tal et al. (2013); Wang et al. (2015); Jiang and Guan (2015); Hu and…”
Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization (2020, neutral): “[21] Arnab Nilim and Laurent El Ghaoui.”
Actor-Critic Algorithms for Risk-Sensitive MDPs (2013, background): “, [12, 7, 24]), and 2) the inherent uncertainty related to the stochastic nature of the system, which is the topic of risk-sensitive MDPs (e.”
Distributionally Robust Markov Decision Processes (2010, methodology): “Throughout this paper we make a standard assumption (cf [5] [6] [8]) that P i s is nonempty, convex and compact.”
Tractable Objectives for Robust Policy Optimization (2012, background): “One formulation for parameter uncertainty assumes that the parameters are taken from uncertainty sets R ∈ R and P ∈ P [12].”
Optimization for machine learning (2010, neutral): “To assess a strategy we use the expected cumulative regret defined as: Rn = n max 1≤i≤K µi − n∑ t=1 EµIt .”
Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, background): “Early studies [183], [184] focus on robust MDP with uncertain parameters.”
Robust and Data-Driven Optimization: Modern Decision-Making Under Uncertainty (2006, methodology): “In the mid-1990s, research teams led by Ben-Tal and Nemirovski ([7], [8], [9]) and El-Ghaoui and Lebret ([27], [28]) addressed the issue of overconservatism by restricting the uncertain parameters to belong to ellipsoidal uncertainty sets, which removes the most unlikely outcomes from consideration ”
Scaling up budgeted reinforcement learning (2019, background): “…risk-averse variants where the optimisation criteria include other statistics of the performance, such as the worst-case realisation [Iyengar, 2005; Nilim and El Ghaoui, 2005; Wiesemann et al. , 2013], the variance-penalised expectation [Garc´ıa and Fern´andez, 2015; Tamar et al. , 2012], or the…”
Robust Policy Optimization with Baseline Guarantees (2015, methodology): “We now summarize the standard results of robust value iteration from [16, 9].”
Towards Verified Artificial Intelligence (2016, background): “For example, the formalism of convex Markov decision processes (convex MDPs) [37, 39, 44] provide a way of representing uncertainty in the values of learned transition probabilities.”