Synthesizing Neural Network Controllers with Probabilistic Model-Based Reinforcement Learning

By J. A. G. Higuera, David Meger, G. Dudek. 2018

41 citations recorded by the source corpus.

Ahstract- We present an algorithm for rapidly learning neural network policies for robotics systems. The algorithm follows the model-based reinforcement learning paradigm and improves upon existing algorithms: PILeO and a sample-based version of PILeo with neural network dynamics (Deep-PILeO). To improve convergence, we propose a model-based algorithm that uses fixed random numbers and clips gradients during optimization. We propose training a neural network dynamics model using variational dropout with truncated Log-Normal noise. These improvements enable data-efficient synthesis of complex neural network policies. We test our approach on a variety of benchmark tasks, demonstrating data-efficiency that is competitive with that of PILeO, while being able to optimize complex neural network controllers. Finally, we assess the performance of the algorithm for learning motor controllers for a six legged autonomous underwater vehicle. This demonstrates the potential of the algorithm for scaling up the dimensionality and dataset sizes, in more complex tasks.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, methodology): “Previous work in model-based reinforcement learning has focused on planning in low-dimensional state spaces (Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Chua et al., 2018), combining the beneﬁts of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et…”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “This is used to combat the well-known problems with compounding model errors, and is present in policy search methods (Deisenroth & Ras-mussen, 2011; Gal et al., 2016; Higuera et al., 2018), combined with trajectory optimization (Chua et al., 2018) or value-based methods (Buckman et al., 2018).”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “Such probabilistic NNs have previously been used for control, including using dropout [12], [41] and α-divergence [13].”
Calibrated Model-Based Deep Reinforcement Learning (2019, background): “Although deep neural networks can signiﬁcantly improve model-based planning algorithms (Higuera et al., 2018; Chua et al., 2018), their estimates of predictive uncertainty are often inaccurate (Guo et al., 2017a; Kuleshov et al., 2018).”
Mastering Atari with Discrete World Models (2020, background): “World models Several model-based agents focus on proprioceptive inputs (Watter et al., 2015; Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Chua et al., 2018; Wang et al., 2019; Wang and Ba, 2019), model images without using them for planning (Oh et al., 2015; Krishnan et al., 2015;…”