Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias

By Abhinav Kumar Gupta, Adithyavairavan Murali, Dhiraj Gandhi, Lerrel Pinto. 2018

166 citations recorded by the source corpus.

Data-driven approaches to solving robotic tasks have gained a lot of traction in recent years. However, most existing policies are trained on large-scale datasets collected in curated lab settings. If we aim to deploy these models in unstructured visual environments like people's homes, they will be unable to cope with the mismatch in data distribution. In such light, we present the first systematic effort in collecting a large dataset for robotic grasping in homes. First, to scale and parallelize data collection, we built a low cost mobile manipulator assembled for under 3K USD. Second, data collected using low cost robots suffer from noisy labels due to imperfect execution and calibration errors. To handle this, we develop a framework which factors out the noise as a latent variable. Our model is trained on 28K grasps collected in several houses under an array of different environmental conditions. We evaluate our models by physically executing grasps on a collection of novel objects in multiple unseen homes. The models trained with our home dataset showed a marked improvement of 43.7% over a baseline model trained with data collected in lab. Our architecture which explicitly models the latent noise in the dataset also performed 10% better than one that did not factor out the noise. We hope this effort inspires the robotics community to look outside the lab and embrace learning based approaches to handle inaccurate cheap robots.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Robotics & Control

Citations in the corpus

RoboNet: Large-Scale Multi-Robot Learning (2019, background): “Large-scale, self-supervised robot learning approaches have adopted a similar viewpoint [24, 25, 26, 9, 15, 10, 27, 7].”
Visual Imitation Made Easy (2020, background): “Resulting in-lab demonstrations often contain little to no variability in objects or environments, which severely limits the generalizability of the learned skills in novel, previously unseen situations [8].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, neutral): “These include Robot Learning in Homes [21], Meta-World [61], the ProcGen benchmark [10].”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “There is a long history of robotics benchmarks [28], datasets [29, 30, 31, 32, 33, 34, 35], competitions [36] and standardized object sets [37, 38] that have played an important role in robotics research.”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “These problems have manifested in many ways, including shifts induced by weather and lighting changes (Wulfmeier et al., 2018), location changes (Gupta et al., 2018), and the simulation-to-real-world gap (Sadeghi and Levine, 2017; Tobin et al., 2017).”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “This includes works that generalize skills to novel objects [31, 32, 33, 34, 35], to novel 76 environments [36], from simulation to reality [37, 38, 39, 40, 41], and to new manipulation skills 77 and objects [42, 43, 44, 45].”
A Survey of Deep Learning Applications to Autonomous Vehicle Control (2019, background): “For complex tasks, such as autonomous driving, the diversity of the data set should be ensured if the aim is to train a generalisable model which can drive in all different environments [56], [57].”
AllenAct: A Framework for Embodied AI Research (2020, unknown): “…[12, 13, 14], question answering [15, 16], task completion [17], instruction following [18, 19], language grounding [20, 21], grasping [22, 23], object manipulation [24, 25], future prediction [26], and multi-agent collaboration [27, 28]; as well as using interaction as a tool to learn:…”
PyRobot: An Open-source Robotics Framework for Research and Benchmarking (2019, methodology): “The ﬁeld of data-driven robotics has also seen tremendous excitement and energy in the past several years [14, 15, 26, 31, 34, 39, 40, 43, 53–55 , 65 ].”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, unknown): “Broadly, these can be categorized into datasets collected autonomously via scripted and semi-random behaviors or learned agents [3, 8, 20, 26, 27, 32, 40], and datasets collected via human teleoperation [1, 2, 13, 14, 24, 36, 50, 59].”
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? (2022, background): “, 2009) has been a popular and very effective embedding model for extracting visual features for down-stream robotic manipulation tasks (Gupta et al., 2018; Mandlekar et al., 2020; Shah and Kumar, 2021).”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, background): “A significant body of work deals with learning policies and predictive models for robotic grasping [53, 36, 46, 14, 64], with the aim of generalizing to new objects.”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, background): “…depth images or object bounding boxes, imitation learning allows mobile manipulators to learn end-to-end by directly mapping raw RGB observations to whole-body actions, showing promising results through large-scale training using real-world data [4, 12, 78] in indoor environments [39, 78].”
RUBi: Reducing Unimodal Biases in Visual Question Answering (2019, unknown): “For instance, some methods are focused on gender biases [33, 34], some others on the human reporting biases [35], and also on the shift in distribution between lab-curated data and real-world data [36].”
π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, unknown): “More recently, large-scale datasets collected across many domains [41, 68, 63, 67, 14, 49] have been shown to enable generalization of simple but end-to-end learned tasks to new environments [33, 31, 67, 69, 26, 49, 28, 64].”
Learning to Manipulate Deformable Objects without Demonstrations (2019, methodology): “From grasping objects in clutter [56, 44, 31, 27, 12] to dexterous in-hand manipulation of objects [1, 69], modern robotic algorithms have transformed object manipulation.”