Model-based reinforcement learning with parametrized physical models and optimism-driven exploration

By Christopher Xie, Sachin Patil, Teodor Mihai Moldovan, Sergey Levine, Pieter Abbeel. 2015

53 citations recorded by the source corpus.

In this paper, we present a robotic model-based reinforcement learning method that combines ideas from model identification and model predictive control. We use a feature-based representation of the dynamics that allows the dynamics model to be fitted with a simple least squares procedure, and the features are identified from a high-level specification of the robot's morphology, consisting of the number and connectivity structure of its links. Model predictive control is then used to choose the actions under an optimistic model of the dynamics, which produces an efficient and goal-directed exploration strategy. We present real time experimental results on standard benchmark problems involving the pendulum, cartpole, and double pendulum systems. Experiments indicate that our method is able to learn a range of benchmark tasks substantially faster than the previous best methods. To evaluate our approach on a realistic robotic control task, we also demonstrate real time control of a simulated 7 degree of freedom arm.

Subfields: Artificial Intelligence, Control and Systems Engineering, Molecular Biology, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Molecular Biology

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “Another, related line of work involves learning models of the environment (Schmidhuber, 2010; Xie et al., 2015; Oh et al., 2015).”
Toward Causal Representation Learning (2021, background): “Particularly relevant for causal leaning are generative world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see I NTRODUCTION above) to train RL agents [48], [99], [127], [178], [214], [231], [248], [268], [271].”
Towards Causal Representation Learning (2021, background): “world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see INTRODUCTION above) to train RL agents [127, 248, 98, 47, 271, 178, 232, 214, 268].”
Shaping Belief States with Generative Environment Models for RL (2019, background): “The idea of learning general world models to support decision-making is probably one of the most pervasive ideas in AI research, [30, 11, 41, 42, 14, 43, 44, 45, 46, 33, 30, 47, 48, 3, 49, 50, 51, 52].”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “Prior work in model-based RL mostly quantiﬁes uncertainty with either heuristics or simpler models (Moldovan et al., 2015; Xie et al., 2016; Deisenroth & Rasmussen, 2011a).”
Model-Based Reinforcement Learning with Value-Targeted Regression (2020, background): “where the 6 regression targets are often xed objects (such as raw observations, features or keypoints; e.g. Jaksch et al. (2010); Osband & Van Roy (2014); Abbasi-Yadkori & Szepesvari (2015); Xie et al. (2016); Agrawal & Jia (2017); Yang & Wang (2019a); Kaiser et al. (2019)). For a co”
Towards Cognitive Exploration through Deep Reinforcement Learning for Mobile Robots (2016, unknown): “Conventional reinforcement learning methods improved the controller performances in path-planning of robot-arms [16] and controlling of helicopters [17].”
Exploration in Deep Reinforcement Learning: A Survey (2022, methodology): “The linear Gaussian algorithm can also be used as a model of the reward [110].”
The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving (2019, methodology): “This approach is often used when applying model predictive control (Feldbaum, 1960; C. Xie, Patil, Moldovan, Levine, & Abbeel, 2016; Fu, Levine, & Abbeel, 2016; Janner et al., 2018; Deisenroth & Rasmussen, 2011), also often called system identiﬁcation (Chiuso & Pillonetto, 2019).”
Learning-Based Model Predictive Control for Safe Exploration (2018, methodology): “, in exploration approaches based on the optimism in the face of uncertainty principle [2].”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, background): “…recent papers have explored the idea of incorporating physics priors into learning dynamics models of environments (Nguyen-Tuong and Peters, 2010; Cutler et al., 2014; Cutler and How, 2015; Scholz et al., 2014; Kansky et al., 2017; Battaglia et al., 2016; Mrowca et al., 2018; Xie et al., 2016).”
Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains (2016, methodology): “Another approach is to learn a dynamics model and to then use either optimistic estimates [Xie et al. , 2015] or uncertainty [Stadie et al. , 2015] in the model to provide exploration bonuses (see also [Guez et al. , 2012]).”