Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols

By Berk Çalli, Aaron Walsman, Arjun Singh, Siddhartha S. Srinivasa, Pieter Abbeel, Aaron M. Dollar. 2015

326 citations recorded by the source corpus.

In this paper we present the Yale-CMU-Berkeley (YCB) Object and Model set, intended to be used to facilitate benchmarking in robotic manipulation, prosthetic design and rehabilitation research. The objects in the set are designed to cover a wide range of aspects of the manipulation problem; it includes objects of daily life with different shapes, sizes, textures, weight and rigidity, as well as some widely used manipulation tests. The associated database provides highresolution RGBD scans, physical properties, and geometric models of the objects for easy incorporation into manipulation and planning software platforms. In addition to describing the objects and models in the set along with how they were chosen and derived, we provide a framework and a number of example task protocols, laying out how the set can be used to quantitatively evaluate a range of manipulation approaches including planning, learning, mechanical design, control, and many others. A comprehensive literature survey on existing benchmarks and object datasets is also presented and their scope and limitations are discussed. The set will be freely distributed to research groups worldwide at a series of tutorials at robotics conferences, and will be otherwise available at a reasonable purchase cost. It is our hope that the ready availability of this set along with the ground laid in terms of protocol templates will enable the community of manipulation researchers to more easily compare approaches as well as continually evolve benchmarking tests as the field matures.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Industrial and Manufacturing Engineering

Citations in the corpus

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “There is a long history of robotics benchmarks [28], datasets [29, 30, 31, 32, 33, 34, 35], competitions [36] and standardized object sets [37, 38] that have played an important role in robotics research.”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, methodology): “[15] propose the YCB dataset with real objects and their 3D scans.”
TriFinger: An Open-Source Robot for Learning Dexterity (2020, background): “This lack of standardized real-world benchmarks has been recognized by the robotics and reinforcement learning community (Behnke, 2006; Bonsignorio & del Pobil, 2015; Calli et al., 2015a;b; Amigoni et al., 2015; Murali et al., 2019a).”
Dex-Net 1.0: A cloud-based network of 3D objects for robust grasp planning using a Multi-Armed Bandit model with correlated rewards (2016, methodology): “0 contains 13,252 3D mesh models: 8,987 from the SHREC 2014 challenge dataset [28], 2,539 from ModelNet40 [45], 1,371 from 3DNet [44], 129 from the KIT object database∗ [19], 120 from BigBIRD∗ [38], 80 from the Yale-CMU-Berkeley dataset∗ [8], and 26 from the Amazon”
Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization (2021, methodology): “…STL-10 (Coates et al., 2011), ImageNet (Deng et al., 2009), ImageNet-V2 (Recht et al., 2019)), – a pose estimation testbed based on YCB-Objects (Calli et al., 2015), – and two distribution shifts derived from concrete applications of image classiﬁcation: satellite imagery and wildlife photos…”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “7), but existing datasets with high-resolution object meshes do not include 2D images samples [1, 9].”
Objaverse: A Universe of Annotated 3D Objects (2022, background): “Datasets such as KIT [33], YCB [5], BigBIRD [64], IKEA [40], and Pix3D [66] provide image-calibrated models over a diverse set of household objects, but severely lack in scale with only a few hundred objects at most.”
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments (2023, methodology): “To not restrict to either possibility, we facilitate scene designing by using graphical interfaces and also providing tools for importing different datasets [32], [33], [34].”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “Datasets such as NYUDepth v2 [19] lack annotations for precise pose of each individual objects, while other datasets such as LabelFusion [16], YCB dataset [3] and REAL275 [29] annotate monocular RGBD images of opaque objects.”