Collective robot reinforcement learning with distributed asynchronous guided policy search

By Ali Yahya, Adrian Shuai Li, Mrinal Kalakrishnan, Yevgen Chebotar, Sergey Levine. 2016

163 citations recorded by the source corpus.

Policy search methods and, more broadly, reinforcement learning can enable robots to learn highly complex and general skills that may allow them to function amid the complexity and diversity of the real world. However, training a policy that generalizes well across a wide range of real-world conditions requires far greater quantity and diversity of experience than is practical to collect with a single robot. Fortunately, it is possible for multiple robots to share their experience with one another, and thereby, learn a policy collectively. In this work, we explore distributed and asynchronous policy learning as a means to achieve generalization and improved training times on challenging, real-world manipulation tasks. We propose a distributed and asynchronous version of guided policy search and use it to demonstrate collective policy learning on a vision-based door opening task using four robots. We describe how both policy learning and data collection can be conducted in parallel across multiple robots, and present a detailed empirical evaluation of our system. Our results indicate that distributed learning significantly improves training time, and that parallelizing policy learning and data collection substantially improves utilization. We also demonstrate that we can achieve substantial generalization on a challenging real-world door opening task.

Subfields: Artificial Intelligence, Control and Systems Engineering, Mechanical Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Mechanical Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “In this context, because neural network training is often slow, decoupling allows us to make many parameter update steps per step in the environment, ensuring that the networks are well ﬁt to the data that is currently available.”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, neutral): “This result agrees with a concurrent result on parallelized guided policy search [35].”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “, Atari [29]), this is significantly more challenging in real-world physical tasks, as it requires human intervention in the form of manual resets or the availability of a hard coded reset [28, 47, 37, 52, 21].”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, background): “While there have been recent attempts on learning by collecting large scale data directly on real robots [1, 2, 3, 4], such an approach still remains challenging as collecting real world data is prohibitively laborious and expensive.”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, methodology): “Reinforcement learning offers a promising avenue for tackling this problem, but current work on reinforcement learning tackles the problem of mastering individual skills, such as hitting a ball [1], opening a door [2, 3], or throwing [4].”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “The model is close to [1]–[3] with the key architectural traits of having a few convolutional layers followed by a channel-wise spatial expectation operation, which has been widely used [10], [11], [25], [29], [36]–[38].”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “For example, using thermal cameras for tracking fluids [35], mocap sensors [21] or computer vision systems [34] for tracking objects, and accelerometers for determining the state of a door [44].”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, methodology): “This result suggests that our model can learn end-to-end visuomotor policies without a pretraining phase or the need of auxiliary tasks, as opposed to previous work on visuomotor learning [3, 23, 52].”
Deep predictive policy training using reinforcement learning (2017, methodology): “The encoding inherently preserves spatial distances in the input image and is therefore suitable for robotic manipulation tasks [6], [12], [16], [18].”
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias (2018, methodology): “Similarly, Nair et al. [10] collects data for manipulating a rope on a table while Yahya et al. [27] used several robots in parallel to train a policy to open a door.”
Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control (2017, background): “Despite many recent advances in RL [24, 39, 44], a main limitation of current RL algorithms remains its data inefﬁciency, i.e., the required number of interactions with the environment is impractically high.”
Motion Planning Networks: Bridging the Gap Between Learning-Based and Classical Motion Planners (2019, methodology): “DRL has solved various challenging robotic tasks using both model-based [28]–[30] and model-free [31]–[33] approaches.”
SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark (2018, methodology): “Model-based methods [7, 8, 38] often enjoy sample efficiency of learning, but pose significant challenges of generalization due to their strong model assumptions.”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, background): “Robotic RL. RL has been applied to a wide variety of robotic manipulation tasks, including grasping objects [11, 12], in-hand object manipulation [13, 14, 15, 16], pouring ﬂuids [17], door opening [18, 19], and manipulating cloth [20, 21].”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, background): “typically requires highly instrumented, and sometimes creative, setups such as motion capture and object tracking [13], [14], accelerometers to detect door opening [15], or thermal cameras to detect pouring [16].”
Benchmarking Reinforcement Learning Algorithms on Real-World Robots (2018, background): “ate information or auxiliary task-speciﬁc steps and knowledge (e.g. Levine et al. 2016, Riedmiller et al. 2018), (b) preparation in simulation (e.g. Rusu et al. 2017) (c) collaborative learning (e.g. Yahya et al. 2017), and (d) learning from demonstrations (e.g. Hester et al. 2017). However, reinfor”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “…application area in which HiL methods could have signiﬁcant real-world impact is robotic manipulation, since learning often requires extensive engineering in the real world (Yahya et al., 2017; Schenck & Fox, 2017; Kormushev et al., 2010; Rusu et al., 2017; Akkaya et al., 2019; Peng et al., 2020).”
Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system (2018, unknown): “Directly training on hardware has been feasible in some cases [25], [26], but domains with many dimensions or highly nonlinear dynamics will always require more data to sufﬁciently explore, human demonstrations for imitation learning, and/or parameterized explorations [27], [28], [29], [30].”