Model-Based Policy Gradients with Parameter-Based Exploration by Least-Squares Conditional Density Estimation
By Syogo Mori, Voot Tangkaratt, Tingting Zhao, Jun Morimoto, Masashi Sugiyama. 2013
26 citations recorded by the source corpus.
The goal of reinforcement learning (RL) is to let an agent learn an optimal control policy in an unknown environment so that future expected rewards are maximized. The model-free RL approach directly learns the policy based on data samples. Although using many samples tends to improve the accuracy of policy learning, collecting a large number of samples is often expensive in practice. On the other hand, the model-based RL approach first estimates the transition model of the environment and then learns the policy based on the estimated transition model. Thus, if the transition model is accurately learned from a small amount of data, the model-based approach is a promising alternative to the model-free approach. In this paper, we propose a novel model-based RL method by combining a recently proposed model-free policy search method called policy gradients with parameter-based exploration and the state-of-the-art transition model estimator called least-squares conditional density estimation. Through experiments, we demonstrate the practical usefulness of the proposed method.
Subfields: Artificial Intelligence, Computational Theory and Mathematics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tengyu Ma
- Yuanzhi Li
- Sergey Levine
- Yuandong Tian
- Trevor Darrell
- Jason D. Lee
- Simon Shaolei Du
- Huazhe Xu
In Computational Theory and Mathematics
- Jean-Baptiste Mouret
- Vassilis Vassiliades
- Katsu Yamane
- Trevor Darrell
- Tengyu Ma
- Konstantinos Chatzilygeroudis
- Yuanzhi Li
- Sehoon Ha
Citations in the corpus
- Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “…2013; Lioutikov et al., 2014; Levine & Abbeel, 2014; Yip & Camarillo, 2014), mixture of Gaussians (Khansari-Zadeh & Billard, 2011), to neural networks (Hunt et al., 1992; Nagabandi et al., 2017; Kurutach et al., 2018; Tangkaratt et al., 2014; Sanchez-Gonzalez et al., 2018; Pascanu et al., 2017).”
- Reducing hardware experiments for model learning and policy optimization (2015, background): “Instead, Tangkaratt et al. [16] used least-squares conditional density estimation (LSCDE) to learn the dynamics model without Gaussian assumption on the transitions.”
- Black-box data-efficient policy search for robotics (2017, methodology): “Both the model-based PGPE [30] and the PILCO [13] algorithm use gradient-based policy updates.”