GRAB: A Dataset of Whole-Body Human Grasping of Objects

By Omid Taheri, Nima Ghorbani, Michael J. Black, Dimitrios Tzionas. 2020

571 citations recorded by the source corpus.

Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time. While "grasping" is commonly thought of as a single hand stably lifting an object, we capture the motion of the entire body and adopt the generalized notion of "whole-body grasps". Thus, we collect a new dataset, called GRAB (GRasping Actions with Bodies), of whole-body grasps, containing full 3D shape and pose sequences of 10 subjects interacting with 51 everyday objects of varying shape and size. Given MoCap markers, we fit the full 3D body shape and pose, including the articulated face and hands, as well as the 3D object pose. This gives detailed 3D meshes over time, from which we compute contact between the body and object. This is a unique dataset, that goes well beyond existing ones for modeling and understanding how humans grasp and manipulate objects, how their full body is involved, and how interaction varies with the task. We illustrate the practical value of GRAB with an example application; we train GrabNet, a conditional generative network, to predict 3D hand grasps for unseen 3D object shapes. The dataset and code are available for research purposes at this https URL.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Evaluation & Benchmarks, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Evaluation & Benchmarks

In Robotics & Control

Citations in the corpus

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “VISOR is also related to efforts that capture hand-object relations, including for identifying active objects [35, 4], next active objects [33, 25], object articulation [44] and grasping [18, 39, 38, 40].”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “A line of works [2,7,21,23,67] have also attempted to generate hand pose conditioned on objects.”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, methodology): “Finally, the most related approaches to our work was presented in [33] and [34], where neural networks are used to predict hand parameters of the MANO hand model [35] given object information.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, unknown): “[21, 39, 46, 49, 56, 57] all use analytically synthesized datasets from the GraspIt! simulator [63], whereas [7, 9, 81] use datasets of captured”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, methodology): “Several applications have emerged on conditional scene generation [24, 47, 71, 81, 88], human pose [16, 32, 87, 90, 92] and motion generation [14, 32, 49, 60, 68–70, 73] in furnished 3D indoor scenes, and object-conditioned grasp pose generation [25, 31, 34, 62, 78].”
Hand-Object Contact Consistency Reasoning for Human Grasps Generation (2021, methodology): “Table 4 shows the performance of the GraspCVAE trained by losses we proposed and losses from [3], which are tested on the Obman test set.”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Another line of datasets focus on hand-object contact reasoning [12, 17, 78, 14].”
HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction (2022, background): “Understanding human-object interaction has long been a pursuit for computer vision researchers and many previous efforts have been focusing on constructing third-person view datasets [5, 8, 38, 40].”
Effective Whole-body Pose Estimation with Two-stages Distillation (2023, background): “25, 31, 65], human-object interaction [7, 42], and pose-conditioned human image and motion generation [10, 24, 27, 59].”