Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set

By Berk Çalli, Aaron Walsman, Arjun Singh, Siddhartha S. Srinivasa, Pieter Abbeel, Aaron M. Dollar. 2015

600 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Industrial and Manufacturing Engineering

Citations in the corpus

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Among those, many object datasets are designed for benchmarking robot manipulation [6, 23, 33, 51].”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, methodology): “For grasping, both OpenGrasp [36] and VisGraB [37] are popular simulation-based benchmarks, whilst the YCB dataset [38] focuses on real-world objects.”
Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items (2022, methodology): “The YCB (YaleCMU-Berkeley) Dataset [16] is 77 real-world graspable objects and parallel model database of textured meshes to facilitate sim-to-real domain adaptation.”
A Survey on Active Simultaneous Localization and Mapping: State of the Art and New Frontiers (2022, background): “In recent years, a significant effort has been made in robotics to address challenges in benchmarking [267] and reproducibility [268].”
CosyPose: Consistent multi-view multi-object 6D pose estimation (2020, background): “The objects are picked from a subset of 21 objects from the YCB object set [18] for which reconstructed 3D models are available.”
PyRobot: An Open-source Robotics Framework for Research and Benchmarking (2019, methodology): “The YCB dataset went a step further by distributing a physical dataset of household and kitchen objects with corresponding meta data (shape, RGBD scans, etc) [19].”
Yale-CMU-Berkeley dataset for robotic manipulation research (2017, background): “Nevertheless, for detailed information about the YCB benchmarking effort in general, we refer the reader to Calli et al. (2015b), which focuses on the criteria for choosing the objects and their utilization for benchmarking in robotics.”
Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach (2018, methodology): “The objects were chosen from the standard robotic grasping datasets the ACRV Picking Benchmark (APB) [16] and the YCB Object Set [5], both of which provide item speciﬁcations and online purchase links.”
Learning robust, real-time, reactive robotic grasping (2019, methodology): “The ACRV Picking Benchmark (APB) (Leitner et al., 2017) and the YCB Object Set (Calli et al., 2015) both define item sets and manipulation tasks.”
TriFinger: An Open-Source Robot for Learning Dexterity (2020, background): “The lack of standardized real-world benchmarks has been recognized by the robotics and reinforcement learning community [11, 12, 13, 14, 15, 16].”
Learning a Decentralized Multi-arm Motion Planner (2020, methodology): “In each run of our demo environment: (1) grasp objects are randomly sampled from a YCB dataset [37] subset consisting of 6 objects plus a cube, and (2) the initial joint conﬁgurations for each arm is uniformly randomized in the range [-0.4rad, 0.4rad] from a home conﬁguration.”
Cartman: The Low-Cost Cartesian Manipulator that Won the Amazon Robotics Challenge (2017, background): “Datasets and benchmarks are increasingly exploited in the robotics community to find solutions for these issues [8], [9], yet the applied nature of competitions makes them one of the greatest drivers of progress – from self-driving cars, to humanoids, to robotic picking.”
The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research (2016, background): “While these focus more on simulation, the YCB [11] dataset defines a set of (physical) objects that cover a wide range of aspects of the manipulation problem, including size, weights and rigidity.”