Agnostic System Identification for Model-Based Reinforcement Learning

By Stéphane Ross, Drew Bagnell. 2012

157 citations recorded by the source corpus.

A fundamental problem in control is to learn a model of a system from observations that is useful for controller synthesis. To provide good performance guarantees, existing methods must assume that the real system is in the class of models considered during learning. We present an iterative method with strong guarantees even in the agnostic case where the system is not in the class. In particular, we show that any no-regret online learning algorithm can be used to obtain a near-optimal policy, provided some model achieves low training error and access to a good exploration distribution. Our approach applies to both discrete and continuous domains. We demonstrate its efficacy and scalability on a challenging helicopter domain from the literature.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, methodology): “Sorg et al. (2010) and Zucker and Bagnell (2012) derived complex policies by adapting a reward function for simple optimal control problems using policy-search techniques.”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “The work of Ross and Bagnell [32] considered a straightforward approach of learning a model from offline data, followed by planning.”
Bootstrapping with Models: Confidence Intervals for Off-Policy Evaluation (2016, neutral): “from a given exploration distribution [14].”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, background): “This straightforward approach is known to fail in the offline RL setting, both in theory and practice, due to distribution shift and model-bias [51, 26].”
Preparing for the Unknown: Learning a Universal Policy with Online System Identification (2017, background): “Ross and Bagnell [28] provided a proof that such iterative processes can converge to an optimal policy, given an accurate dynamic model learning method and a good policy search algorithm.”
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles (2016, methodology): “Standard model-based RL methods typically operate by finding a maximum-likelihood estimate to the target dynamics model [10, 11], followed by policy optimization.”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “Nominal dynamics models based on knowledge of physics [1], or through learning [2], complements a growing body of work on successful simulation to reality transfer and system identification [3, 4, 5, 6, 7].”
Importance Sampling Policy Evaluation with an Estimated Behavior Policy (2018, background): “In these settings, an alternative to importance sampling policy evaluation is model-based policy evaluation [20, 16, 9].”
Self-Correcting Models for Model-Based Reinforcement Learning (2016, methodology): “We now analyze H-DAgger-MC, adapting Ross and Bagnell (2012)’s DAgger analysis.”
Instabilities of Offline RL with Pre-Trained Neural Representation (2021, methodology): “Furthermore, there are number of oﬄine RL methods with promising performance on certain benchmark tasks [Laroche et al., 2019, Fujimoto et al., 2019, Jaques et al., 2020, Kumar et al., 2019, Agarwal et al., 2020, Wu et al., 2020, Kidambi et al., 2020, Ross and Bagnell, 2012].”
High Confidence Off-Policy Evaluation with Models (2016, methodology): “We will make use of the following fact from Ross and Bagnell [13]:”
Efficient Online Reinforcement Learning with Offline Data (2023, methodology): “We call this ‘symmetric sampling’, whereby for each batch we sample 50% of the data from our replay buffer, and the remaining 50% from the offline data buffer, resembling the scheme used by Ross & Bagnell (2012).”
Grounded Action Transformation for Robot Learning in Simulation (2017, methodology): “We have started to explore methods similar to the Dataset Aggregation (DAgger) algorithm (Ross and Bagnell 2012) which collects new data to retrain f − 1 sim as the state distribution of θ in E sim changes with grounding.”
Guided Policy Search (2013, methodology): “Numerous recent methods have proposed to learn the model (Abbeel et al., 2006; Deisenroth & Rasmussen, 2011; Ross & Bagnell, 2012), and if we use initial examples , only local models are required.”
Improved Learning of Dynamics Models for Control (2016, methodology): “[18] provide theoretical guarantees for this algorithm, as long as we also sample states from the exploration distribution”