Reinforcement learning with misspecified model classes

By Joshua Mason Joseph, Alborz Geramifard, John W. Roberts, Jonathan P. How, Nicholas A. Roy. 2013

43 citations recorded by the source corpus.

Real-world robots commonly have to act in complex, poorly understood environments where the true world dynamics are unknown. To compensate for the unknown world dynamics, we often provide a class of models to a learner so it may select a model, typically using a minimum prediction error metric over a set of training data. Often in real-world domains the model class is unable to capture the true dynamics, due to either limited domain knowledge or a desire to use a small model. In these cases we call the model class misspecified, and an unfortunate consequence of misspecification is that even with unlimited data and computation there is no guarantee the model with minimum prediction error leads to the best performing policy. In this work, our approach improves upon the standard maximum likelihood model selection metric by explicitly selecting the model which achieves the highest expected reward, rather than the most likely model. We present an algorithm for which the highest performing model from the model class is guaranteed to be found given unlimited data and computation. Empirically, we demonstrate that our algorithm is often superior to the maximum likelihood learner in a batch learning setting for two common RL benchmark problems and a third real-world system, the hydrodynamic cart-pole, a domain whose complex dynamics cannot be known exactly.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Value Iteration Networks (2016, neutral): “In addition, the effects of modelling errors in VINs can be mitigated by training the network end-to-end, similarly to the methods in [13, 11].”
Agnostic System Identification for Monte Carlo Planning (2015, background): “Sorg, Lewis, and Singh (2010) and Joseph et al. (2013) both address the agnostic setting by adapting model parameters to directly improve agent performance (rather than model accuracy) via policy gradient algorithms.”
Iterative Value-Aware Model Learning (2018, methodology): “This type of error propagation analysis has been performed before by Munos [2007], Antos et al.”
Time-regularized interrupting options (2014, unknown): “This is termed model misspecification (Joseph et al., 2013).”
Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning (2017, background): “In safe RL numerous other approaches exists for safe exploration (Joseph et al., 2013; Hans et al., 2008; Garcia & Fernández, 2012; Berkenkamp et al., 2017).”
Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables (2017, background): “In the domain of model-based RL the epistemic uncertainty is the source of model bias (or representational bias, see e.g. Joseph et al. (2013)).”
Model Regularization for Stable Sample Rollouts (2014, background): “Joseph et al. (2013) make a similar observation regarding the dynamics model and use policy gradient to learn model parameters that lead to good planning, rather than low prediction error (demonstrating that the two do not always coincide).”
Decomposition of Uncertainty for Active Learning and Reliable Reinforcement Learning in Stochastic Systems (2017, methodology): “In the batch reinforcement learning setting, Joseph et al. [2013] address model (representational) bias by avoiding building a model all together; instead, they generate pseudo on-policy trajectories using model-free Monte Carlo, a form of local interpolation between real samples from the batch.”
Self-Correcting Models for Model-Based Reinforcement Learning (2016, background): “For instance, Sorg, Lewis, and Singh (2010) and Joseph et al. (2013) both pointed out that the most accurate model by this measure is not necessarily the best for planning.”
The Value Equivalence Principle for Model-Based Reinforcement Learning (2020, methodology): “[22] note that minimizing prediction error is not the same as maximizing the performance of the resulting policy, and propose an algorithm that optimizes the parameters of the model rather than the policy’s.”
Bayesian Learning of Recursively Factored Environments (2013, background): “Recently, Joseph et al. (2013) argued that, when no member of the model class can represent the true environment, a mismatch arises between model accuracy and model usefulness: the best policy may not rely on the most accurate model.”