Optimism-driven exploration for nonlinear systems

By Teodor Mihai Moldovan, Sergey Levine, Michael I. Jordan, Pieter Abbeel. 2015

37 citations recorded by the source corpus.

Tasks with unknown dynamics and costly system interaction time present a serious challenge for reinforcement learning. If a model of the dynamics can be learned quickly, interaction time can be reduced substantially. We show that combining an optimistic exploration strategy with model-predictive control can achieve very good sample complexity for a range of nonlinear systems. Our method learns a Dirichlet process mixture of linear models using an exploration strategy based on optimism in the face of uncertainty. Trajectory optimization is used to plan paths in the learned model that both minimize the cost and perform exploration. Experimental results show that our approach achieves some of the most sample-efficient learning rates on several benchmark problems, and is able to successfully learn to control a simulated helicopter during hover and autorotation with only seconds of interaction time. The computational requirements are substantial.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, background): “Model-based algorithms have explored a variety of dynamics estimation schemes, including Gaussian processes [20], mixture models [21], and local linear system estimation [22], with a more detailed overview in a recent survey [23].”
Model-based reinforcement learning with parametrized physical models and optimism-driven exploration (2015, methodology): “1 Previous work used Bayesian models to accurately estimate this uncertainty [21].”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “models is difﬁcult — even without the RL components, it is an active but widely-open research area. Prior work in model-based RL mostly quantiﬁes uncertainty with either heuristics or simpler models (Moldovan et al., 2015; Xie et al., 2016; Deisenroth & Rasmussen, 2011a). Previous theoretical wo”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, background): “On simple, low-dimensional tasks, model-based approaches have demonstrated remarkable data efﬁciency, learning policies for systems like cart-pole swing-up with under 30 seconds of experience [10, 28].”
SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards (2018, methodology): “This has the interpretation of fitting a locally linear regression to the data (refer to (Moldovan et al. 2015; Khansari-Zadeh and Billard 2011; Kruger et al. 2010; Krishnan* et al. 2015; Murali* et al. 2016) for details).”
A tutorial on task-parameterized movement learning and retrieval (2015, methodology): “, determining the number of Gaussians in the GMM) is compatible with the techniques employed in standard GMM, such as the use of a Bayesian information criterion [82], Dirichlet process [22,50,65,74], iterative pairwise replacement [83], spectral clustering [53, 69,84] or based on segmentation point”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “On simple, lowdimensional tasks, model-based approaches have demonstrated remarkable data efficiency, learning policies for systems like cart-pole swing-up with under 30 seconds of experience (Deisenroth et al., 2014; Moldovan et al., 2015).”