RoboNet: Large-Scale Multi-Robot Learning

By Sudeep Dasari, Frederik D. Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, Chelsea Finn. 2019

437 citations recorded by the source corpus.

Robot learning has emerged as a promising tool for taming the complexity and diversity of the real world. Methods based on high-capacity models, such as deep networks, hold the promise of providing effective generalization to a wide range of open-world environments. However, these same methods typically require large amounts of diverse training data to generalize effectively. In contrast, most robotic learning experiments are small-scale, single-domain, and single-robot. This leads to a frequent tension in robotic learning: how can we learn generalizable robotic controllers without having to collect impractically large amounts of data for each separate experiment? In this paper, we propose RoboNet, an open database for sharing robotic experience, which provides an initial pool of 15 million video frames, from 7 different robot platforms, and study how it can be used to learn generalizable models for vision-based robotic manipulation. We combine the dataset with two different learning algorithms: visual foresight, which uses forward video prediction models, and supervised inverse models. Our experiments test the learned algorithms' ability to work across new objects, new tasks, new scenes, new camera viewpoints, new grippers, or even entirely new robots. In our final experiment, we find that by pre-training on RoboNet and fine-tuning on data from a held-out Franka or Kuka robot, we can exceed the performance of a robot-specific training approach that uses 4x-20x more data. For videos and data, see the project webpage: https://www.robonet.wiki/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Learning Open-World Object Proposals without Learning to Classify (2021, methodology): “Taking a step towards this goal, we investigate the open-world, classagnostic detection ability on highly diverse and hard-to-name objects in RoboNet [10] dataset.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…of robotic grasping, Ebert et al. (2018) propose a model-based algorithm based on prediction of future video frames for learning a variety of robotic manipulation skills from offline data, while Dasari et al. (2019) expand on this approach with a large and diverse dataset of robotic interaction.”
MOPO: Model-based Offline Policy Optimization (2020, background): “Recent advances in machine learning using deep neural networks have shown signiﬁcant successes in scaling to large realistic datasets, such as ImageNet [12] in computer vision, SQuAD [54] in NLP, and RoboNet [9] in robot learning.”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “…in a variety of on-line settings such as games (Silver et al., 2016; Tesauro, 1995; Brown & Sandholm, 2019) and robotics (Levine et al., 2016; Dasari et al., 2019; Peters et al., 2010; Parmas et al., 2019; Pinto & Gupta, 2016; Nachum et al., 2019a), where the agent can act in the environment…”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “[4] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn.”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “In contrast to RoboTurk, we do not crowd source our demonstrations, but instead rely on an inﬁnite supply of generated demonstrations collected via motion planners.”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “This includes works that generalize skills to novel objects [31, 32, 33, 34, 35], to novel environments [36], from simulation to reality [37, 38, 39, 40, 41], and to new manipulation skills and objects [42, 43, 44, 45].”
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets (2021, methodology): “More closely related to our work, RoboNet [6] contains data from multiple robots and domains, but this data is collected using random motions, and does not provide examples of multiple different tasks that can be used for more complex task-directed manipulation.”
An Optimistic Perspective on Offline Reinforcement Learning (2019, background): “This setting can leverage the vast amount of existing logged interactions for real world decision-making problems such as robotics (Cabi et al., 2019; Dasari et al., 2019), autonomous driving (Yu et al.”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “Unlike these works, we don’t make any assumptions about the optimality of the actions in the collected data, allowing the agent to learn from broader offline datasets, including autonomously collected data, which can be considerably easier to collect at scale on a real robot than human tele-operatio”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, positive): “With the exception of RoboNet [23], these datasets contain data of robots of the same type, whereas we focus on data spanning multiple embodiments.”
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning (2021, background): “Similar to before, such work relies on collecting large scale robotic datasets [3, 31, 32], and learning policies that can then be safely deployed after a few steps of finetuning.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “To provide agent-centric training data, RoboNet [25] collects video frames from 7 robot platforms, and Ego4D [43] recruits volunteers to record egocentric videos of household activities.”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “, 2020), recommendation systems (Harper and Konstan, 2015), and robotic manipulation (Finn and Levine, 2017; Sharma et al., 2018; Dasari et al., 2019; Mandlekar et al., 2019), typically in image (or video) format.”
Learning Interactive Real-World Simulators (2023, background): “…5], video captioning and question answering data contain rich high-level descriptions but little low-level movement detail [6, 7], human activity data contains rich human action but little mechanical motion [8, 9], and robotics data contains rich robot action but are limited in quantity [10, 11].”
Transframer: Arbitrary Frame Prediction with Generative Models (2022, methodology): “The sparsity gains of the residual representation are significant in the static-camera RoboNet dataset [12].”
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024, background): “One approach is self-supervised learning, where trial-and-error is used to collect data for tasks like grasping and pushing [24, 36, 18, 19, 46, 7].”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, neutral): “Broadly, these can be categorized into datasets collected autonomously via scripted and semi-random behaviors or learned agents [3, 8, 20, 26, 27, 32, 40], and datasets collected via human teleoperation [1, 2, 13, 14, 24, 36, 50, 59].”