The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research

By Jürgen Leitner, Adam W. Tow, Niko Sünderhauf, Jake E. Dean, Joseph W. Durham, Matthew Cooper, M. Eich, Christopher F. Lehnert, Ruben Mangels, C. McCool and 8 more. 2016

83 citations recorded by the source corpus.

Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark. Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of complete robotic systems — including perception and manipulation — instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.

Subfields: Computer Science Applications, Control and Systems Engineering, Mechanical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

1. Jürgen Leitner
2. Peter Corke
3. Andrew J. Davison
4. Stephen James
5. Douglas Morrison
6. David Rovick Arrojo
7. Zicong Ma
8. Sergey Levine

In Mechanical Engineering

1. Douglas Morrison
2. Peter Corke
3. Andrew J. Davison
4. Stephen James
5. Jürgen Leitner

Citations in the corpus

• RLBench: The Robot Learning Benchmark & Learning Environment (2019, methodology): “The ACRV Picking Benchmark [26] aimed to solve this by creating a similar, but reproducible setup to the ARC.”
• Simitate: A Hybrid Imitation Learning Benchmark (2019, neutral): “Some of the later benchmarks also integrate the replication by actual robotic systems i.e. for grasping [39].”
• Benchmarking Simulated Robotic Manipulation Through a Real World Dataset (2019, background): “c.) [6], [7], motion and grasp planning [8], cooperative or collaborative manipulation [9], and challenging specific sub-tasks (i.e. shelf picking, peg-in-hole, deformable bodies, lego assembly, etc.) [10], [11], [12], [13]. Few simulated benchmarks exist for manipulation, and those available have be”
• Cartman: The Low-Cost Cartesian Manipulator that Won the Amazon Robotics Challenge (2017, methodology): “17 items were used, consisting of 13 items from the Amazon item set and 4 un-seen items from the ACRV picking benchmark set [3].”
• Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach (2018, methodology): “Right: The 12 household objects selected from [5] and [16].”
• Learning robust, real-time, reactive robotic grasping (2019, methodology): “For this paper, we propose a set of 20 reproducible items for testing, comprising 8 3D-printed adversarial objects from Mahler et al. (2017) and 12 items from the APB and YCB object sets, which we believe provide a wide enough range of sizes, shapes, and difficulties to effectively compare results w”