Reinforcement Learning under Model Mismatch
By Aurko Roy, Huan Xu, Sebastian Pokutta. 2017
94 citations recorded by the source corpus.
We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it. We address this problem by extending the framework of robust MDPs to the model-free Reinforcement Learning setting, where we do not have access to the model parameters, but can only sample states from it. We define robust versions of Q-learning, SARSA, and TD-learning and prove convergence to an approximately optimal robust policy and approximate value function respectively. We scale up the robust algorithms to large MDPs via function approximation and prove convergence under two different settings. We prove convergence of robust approximate policy iteration and robust approximate value iteration for linear architectures (under mild assumptions). We also define a robust loss function, the mean squared robust projected Bellman error and give stochastic gradient descent algorithms that are guaranteed to converge to a local minimum.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Dawn Song
- Philipp Krahenbuhl
- Vladlen Koltun
- Paul Trichelair
- Sergey Levine
- Trevor Darrell
- Romain Laroche
- Katelyn Gao
In Computational Theory and Mathematics
- Dawn Song
- Romain Laroche
- Vladlen Koltun
- Philipp Krahenbuhl
- Paul Trichelair
- Keith B. Frikken
- Benny Pinkas
In Management Science and Operations Research
- Dawn Song
- Romain Laroche
- Vladlen Koltun
- Philipp Krahenbuhl
- Paul Trichelair
- Jernej Kos
- Anca Dragan
- Edouard Leurent
Citations in the corpus
- Safe Policy Improvement with Baseline Bootstrapping (2017, background): “(Roy et al., 2017) develop robust versions of the main model-free algorithms, offering algorithms that robustly converge online to the optimal policy.”
- Assessing Generalization in Deep Reinforcement Learning (2018, background): “Nilim & Ghaoui (2004) assume that the set of possible transition probability distribution functions are known, while Lim et al. (2013) and Roy et al. (2017) estimate it using sampled trajectories from the distribution of environments of interest.”