Benchmarking Simulated Robotic Manipulation Through a Real World Dataset

By J. Collins, J. McVicar, Dave Wedlock, Ross Brown, David Howard, Jürgen Leitner. 2019

32 citations recorded by the source corpus.

We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset. Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset. The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies. We apply our benchmark to two simulation environments, i.e., PyBullet and V-Rep, and publish the results. All materials required to benchmark an environment, including protocols and the dataset, can be found at the benchmarks’ website<xref ref-type="fn" rid="fn1">1</xref><fn id="fn1"><label>1</label>[Online]. Available: <uri>https://research.csiro.au/robotics/manipulation-benchmark/</uri>.</fn>.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “Recently there has been an attempt to benchmark the fidelity of simulation environments against the real world [39].”