Using inaccurate models in reinforcement learning

By Pieter Abbeel, Morgan Quigley, Andrew Y. Ng. 2006

272 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Learning Continuous Control Policies by Stochastic Value Gradients (2015, background): “4, the term γV θ arises because the total derivative includes policy gradient contributions from subsequent time steps (full derivation in Appendix A).”
Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search (2018, methodology): “Our CF-GPS algorithm is inspired by and extends work presented by Abbeel et al. (2006) on a method for de-biasing weak models by estimating additive terms in the transition kernel to better match individual, real trajectories.”
Reinforcement learning in robotics: A survey (2013, methodology): “, 2005), steering a RC car with a general analysis of approximate models for policy search in (Abbeel et al., 2006), a pick and place task (Freeman et al.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, background): “This choice is often crucial for an MBRL algorithm, as even small bias can significantly influence the quality of the corresponding controller [Atkeson and Santamaría, 1997, Abbeel et al., 2006].”
Dual Policy Iteration (2018, positive): “In this sense, our analysis is similar in spirit to previous work that uses inaccurate models for policy optimization (An et al., 1988; Abbeel et al., 2006; Kolter & Ng, 2009).”
Model-Ensemble Trust-Region Policy Optimization (2018, methodology): “…learning applications so far were achieved using the simplest possible model parameterization, namely linear models (Bagnell & Schneider, 2001; Abbeel et al., 2006; Levine & Abbeel, 2014; Watter et al., 2015; Levine et al., 2016a; Kumar et al., 2016), where the model either operates directly…”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “Impressive results with model-based RL have been obtained using simple linear models [16, 17, 18, 19].”
Preparing for the Unknown: Learning a Universal Policy with Online System Identification (2017, methodology): “Some widely-used models, such as linear models [3], Gaussian processes [8, 11], and differential equations [35, 4], have proven effective for continuous dynamics and control tasks with relatively low action space.”
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles (2016, methodology): “A number of methods have also been suggested to reduce sample complexity when provided with either a baseline policy [25, 26], expert demonstration [27, 28], or approximate simulator [29, 30].”
Reinforcement learning with multi-fidelity simulators (2014, methodology): “We now instantiate the principles of generating heuristics from low-ﬁdelity simulators and sending learned model data down from high ﬁdelity simulators in the full (multi-state, cumulative reward) RL case.”
TossingBot: Learning to Throw Arbitrary Objects With Residual Physics (2019, background): “ so that human intervention is kept at a minimum. Learning residual models and policies. Our approach to data-efﬁcient learning, Residual Physics, falls under a broader category of hybrid controllers [1,26] that leverage both 1) analytical models to provide initial estimates of control parameters, a”
Self-Correcting Models for Model-Based Reinforcement Learning (2016, unknown): “In addition to assuming a particular form for the planner (one-ply MC with a blind rollout policy), H-DAgger-MC assumes that the model will be “unrolled” (similar to, e.g. Abbeel et al. 2006).”
Data-Efficient Generalization of Robot Skills with Contextual Policy Search (2013, methodology): “Policy search methods are divided into model-free (Williams 1992; Peters and Schaal 2008; Kober and Peters 2010) and modelbased (Bagnell and Schneider 2001; Abbeel et al. 2006; Deisenroth and Rasmussen 2011) approaches.”
Multi-fidelity Gaussian Process Bandit Optimisation (2016, methodology): “Multifidelity methods are used in the robotics community for reinforcement learning tasks by modeling each fidelity as a Markov decision process (Abbeel et al., 2006; Cutler et al., 2014; Kennedy & O’Hagan, 1998).”