Learning model-free robot control by a Monte Carlo EM algorithm

By Nikos A. Vlassis, Marc Toussaint, Georgios D. Kontes, Savas Piperidis. 2009

87 citations recorded by the source corpus.

We address the problem of learning robot control by model-free reinforcement learning (RL). We adopt the probabilistic model for model-free RL of Vlassis and Toussaint (Proceedings of the international conference on machine learning, Montreal, Canada, 2009), and we propose a Monte Carlo EM algorithm (MCEM) for control learning that searches directly in the space of controller parameters using information obtained from randomly generated robot trajectories. MCEM is related to, and generalizes, the PoWER algorithm of Kober and Peters (Proceedings of the neural information processing systems, 2009). In the finite-horizon case MCEM reduces precisely to PoWER, but MCEM can also handle the discounted infinite-horizon case. An interesting result is that the infinite-horizon case can be viewed as a ‘randomized’ version of the finite-horizon case, in the sense that the length of each sampled trajectory is a random draw from an appropriately constructed geometric distribution. We provide some preliminary experiments demonstrating the effects of fixed (PoWER) vs randomized (MCEM) horizon length in two simulated and one real robot control tasks.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Gradient Estimation Using Stochastic Computation Graphs (2015, background): “In fact, similar equations have been derived by interpreting rewards (negative costs) as probabilities, and then taking the variational lower bound on log-probability (e.g., [24]).”
Reinforcement learning in robotics: A survey (2013, methodology): “…flight (Bagnell and Schneider, 2001), learning biped walk patterns (Tedrake, 2004; Tedrake et al., 2005), driving a radiocontrolled (RC) car, learning a jumping behavior for a robot dog (Kolter and Ng, 2009a) (illustrated in Figure 5), and balancing a two-wheeled robot (Vlassis et al., 2009).”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, background): “There is some relationship to the work presented here and work on relative entropy policy search [23], and policy optimization via expectation maximization [35] and KLdivergence [14, 33], however none of these bridge the gap between the two directions of the KLdivergence, nor do they consider any su”
Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, methodology): “We follow the idea of RL as probabilistic inference (39, 71, 72) which leads to a Kullback-Leibler divergence (KL) regularised objective in which the policy Q is regularised against a prior policy P.”
Reinforcement learning of motor skills in high dimensions: A path integral approach (2010, methodology): “In the spirit of these latter ideas, this paper addresses a new method of probabilistic reinforcement learning derived from the framework of stochastic optimal control and path integrals, based on the original work of [14], [4].”
Variational Inference for Policy Search in changing situations (2011, methodology): “Many policy search algorithms (Kober & Peters, 2009; Vlassis et al., 2009; Heidrich-Meisner & Igel, 2009) use inference or inference related methods to iteratively optimize the policy.”
A Survey on Policy Search for Robotics (2013, background): “In most tasks, robots operate in a high-dimensional state space x composed of both internal states (e.g., joint angles, joint velocities, endeffector pose, and body position/orientation) and external states (e.g., object locations, wind conditions, or other robots).”
Policy gradient in Lipschitz Markov Decision Processes (2015, background): “Kober and Peters (2008) and Vlassis et al. (2009) studied policy-search methods based on expectation-maximization.”
Online Model Learning Algorithms for Actor-Critic Control (2015, background): “large effect on this, although more recently expectation-maximisation (EM) methods that work without learning rates have been proposed (Kober and Peters, 2011; Vlassis et al., 2009).”
Variational Policy Search via Trajectory Optimization (2013, unknown): “This is the standard formulation for expectation maximization [9], and has been applied to policy optimization in previous work [8, 21, 3, 11].”
Adaptive Step-Size for Policy Gradient Methods (2013, methodology): “Nonetheless, some policy search methods based on expectation–maximization have been recently proposed; such methods have properties similar to the ones of policy gradients, but the policy update does not require to tune the step size [12, 13].”
Learning to Communicate Implicitly by Actions (2018, methodology): “Solving reinforcement learning as an inference problem has a long history, which dates back to (Kalman, 1960) where the Kalman smoothing is used to solve optimal control in linear dynamics with quadratic cost and has been well studied in (Levine & Koltun, 2013; Vlassis et al., 2009; Kober & Peters, ”