RTMBA: A Real-Time Model-Based Reinforcement Learning Architecture for robot control

By Todd Hester, Michael J. Quinlan, Peter Stone. 2011

88 citations recorded by the source corpus.

Reinforcement Learning (RL) is a paradigm for learning decision-making tasks that could enable robots to learn and adapt to their situation on-line. For an RL algorithm to be practical for robotic control tasks, it must learn in very few samples, while continually taking actions in real-time. Existing model-based RL methods learn in relatively few samples, but typically take too much time between each action for practical on-line learning. In this paper, we present a novel parallel architecture for model-based RL that runs in real-time by 1) taking advantage of sample-based approximate planning methods and 2) parallelizing the acting, model learning, and planning processes in a novel way such that the acting process is sufficiently fast for typical robot control cycles. We demonstrate that algorithms using this architecture perform nearly as well as methods using the typical sequential architecture when both are given unlimited time, and greatly out-perform these methods on tasks that require real-time actions such as controlling an autonomous vehicle.

Subfields: Artificial Intelligence, Automotive Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Control and Systems Engineering

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, background): “Hester et al. (2012) has proposed a real-time architecture for model-based value-function reinforcement learning methods taking into account these challenges.”
Improved Learning of Dynamics Models for Control (2016, background): “Learning based approaches for controlling robotic systems, or more generally autonomous agents, fall into two primary categories: model-based [1–3] and model-free [4–7].”
Safe Exploration of State and Action Spaces in Reinforcement Learning (2012, methodology): “Many RL methods are being used in important and complex tasks (e.g., robot control see Smart & Kaelbling, 2002; Hester, Quinlan, & Stone, 2011, stochastic games see Mannor, 2004; Konen & Bartz-Beielstein, 2009 and control optimization of complex dynamical systems see Salkham, Cunningham, Garg, &…”
Learning from Limited Demonstrations (2013, background): “1 Car Brake Control Simulation In the vehicle brake control simulation [21], the agent’s goal is reach a target velocity, then maintain that target.”
On-line policy learning and adaptation for real-time personalization of an artificial pancreas (2015, background): “For RL algorithms to be practical, they must be able to learn from a handful of samples while continually taking actions in real-time (Hester, Quinlan, & Stone, 2012).”
The Open-Source TEXPLORE Code Release for Reinforcement Learning on Robots (2013, methodology): “The full details of this architecture are described in [7, 9].”
Learning and Using Models (2012, methodology): “TEXPLORE has been used in a gridworld with over 300, 000 states (Hester and Stone, 2010) and run in real-time on an autonomous vehicle (Hester et al, 2011) 4 .”
Reset-free Trial-and-Error Learning for Robot Damage Recovery (2016, methodology): “For example, TEXPLORE has successfully been used to control a real car in real-time [34].”