The Ingredients of Real-World Robotic Reinforcement Learning

By Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, Sergey Levine. 2020

207 citations recorded by the source corpus.

The success of reinforcement learning in the real world has been limited to instrumented laboratory scenarios, often requiring arduous human supervision to enable continuous learning. In this work, we discuss the required elements of a robotic system that can continually and autonomously improve with data collected in the real world, and propose a particular instantiation of such a system. Subsequently, we investigate a number of challenges of learning without instrumentation -- including the lack of episodic resets, state estimation, and hand-engineered rewards -- and propose simple, scalable solutions to these challenges. We demonstrate the efficacy of our proposed system on dexterous robotic manipulation tasks in simulation and the real world, and also provide an insightful analysis and ablation study of the challenges associated with this learning paradigm.

Subfields: Artificial Intelligence, Control and Systems Engineering, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Social Psychology

Citations in the corpus

Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “[54], where the hand must position an object in the center of a bin, starting from any position.”
Reset-Free Lifelong Learning with Skill-Space Planning (2020, background): “Zhu et al. (2020) and Co-Reyes et al. (2020) propose solutions to lack of learning signal in reset-free settings but only consider environments without sink states; the latter requires control over the environment to form a training curriculum.”
Autonomous Reinforcement Learning via Subgoal Curricula (2021, methodology): “Conventional episodic RL algorithms often fail to solve the task entirely in this setting, as shown by Zhu et al. [48] and Figure 2.”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, methodology): “…RL innovations have been demonstrated in pick-and-place problems, addressing challenges such as multi-task learning (114, 115, 125), sample efficiency (54), defining and computing reward (120, 121), resetting the environment (117), and utilizing human demonstrations or offline data (122, 116, 124).”
TriFinger: An Open-Source Robot for Learning Dexterity (2020, background): “There have been isolated successes on real systems (Levine et al., 2018; Zhu et al., 2020; Pinto & Gupta, 2016; Andrychowicz et al., 2020), but these results cannot be compared or reproduced, since each lab has their own robotic setup.”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, background): “Some approaches to real-world robot learning for manipulation utilize RL with real robot data collection or simulation with domain randomization [15, 29, 12, 13, 38, 39, 40, 41, 42, 43], while other approaches focus on imitation learning from expert demonstrations [27, 44, 45, 46, 47, 48].”