Scaling Up Robust MDPs by Reinforcement Learning

By Aviv Tamar, Huan Xu, Shie Mannor. 2013

45 citations recorded by the source corpus.

We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimensionality", MDPs that model real-life problems are typically prohibitively large for such approaches. In this work we employ a reinforcement learning approach to tackle this planning problem: we develop a robust approximate dynamic programming method based on a projected fixed point equation to approximately solve large scale robust MDPs. We show that the proposed method provably succeeds under certain technical conditions, and demonstrate its effectiveness through simulation of an option pricing problem. To the best of our knowledge, this is the first attempt to scale up the robust MDPs paradigm.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Angela P. Schoellig
2. Andreas Krause
3. Felix Berkenkamp
4. Matteo Turchetta
5. Javier García
6. Fernando Fernández
7. Shie Mannor
8. Chen Tessler

In Management Science and Operations Research

1. Fernando Fernández
2. Javier García
3. Chen Tessler
4. Angela P. Schoellig
5. Andreas Krause
6. Shie Mannor
7. Felix Berkenkamp
8. Yonathan Efroni

Citations in the corpus

• Safe Model-based Reinforcement Learning with Stability Guarantees (2017, background): “Similarly, robust MDPs maximize rewards when transition probabilities are uncertain [9, 10].”
• Action Robust Reinforcement Learning and Applications in Continuous Control (2019, background): “Previously, Robust MDPs have been analyzed extensively in the theoretical community, in the tabular case (Nilim & El Ghaoui, 2005; Iyengar, 2005; Xu & Mannor, 2007; Mannor et al., 2012; Wiesemann et al., 2013) and under linear function approximation (Tamar et al., 2013).”
• A comprehensive survey on safe reinforcement learning (2015, background): “…RL                                                                                                                                                   …”