Open-source benchmarking for learned reaching motion generation in robotics
By Andre Lemme, Yaron Meirovitch, Seyed Mohammad Khansari-Zadeh, Tamar Flash, Aude Gemma Billard, Jochen J. Steil. 2015
43 citations recorded by the source corpus.
Abstract This paper introduces a benchmark framework to evaluate the performance of reaching motion generation approaches that learn from demonstrated examples. The system implements ten different performance measures for typical generalization tasks in robotics using open source MATLAB software. Systematic comparisons are based on a default training data set of human motions, which specify the respective ground truth. In technical terms, an evaluated motion generation method needs to compute velocities, given a state provided by the simulation system. This however is agnostic to how this is done by the method or how the methods learns from the provided demonstrations. The framework focuses on robustness, which is tested statistically by sampling from a set of perturbation scenarios. These perturbations interfere with motion generation and challenge its generalization ability. The benchmark thus helps to identify the strengths and weaknesses of competing approaches, while allowing the user the opportunity to configure the weightings between different measures.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering, Evaluation & Benchmarks, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Li Fei-Fei
- Silvio Savarese
- Pieter Abbeel
- Yuke Zhu
- Chelsea Finn
- Trevor Darrell
- Abhinav Kumar Gupta
In Cognitive Neuroscience
- J. Bagnell
- Gerhard Neumann
- Jan Peters
- Yaron Meirovitch
- Sonia Chernova
- Pieter Abbeel
- Roberto Mart'in-Mart'in
- Silvio Savarese
In Control and Systems Engineering
- Yuke Zhu
- Sergey Levine
- Li Fei-Fei
- Silvio Savarese
- Ajay Mandlekar
- J. Bagnell
- Pieter Abbeel
- Jan Peters
In Evaluation & Benchmarks
- Yuke Zhu
- Silvio Savarese
- Li Fei-Fei
- Ajay Mandlekar
- Danfei Xu
- Pieter Abbeel
- Roberto Mart'in-Mart'in
- Jan Peters
In Robotics & Control
- Yuke Zhu
- Pieter Abbeel
- Ajay Mandlekar
- Li Fei-Fei
- Jan Peters
- Gerhard Neumann
- Sergey Levine
- Silvio Savarese
Citations in the corpus
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “Prior work has also established benchmarks for motion-based learning from demonstration meth-ods [71, 72], which directly model entire robot arm trajectories.”
- Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance (2019, methodology): “Overall, despite a common assumption to the contrary, we observe that MSE is not an accurate predictor of generalization performance of a skill learning approach.”
- An Algorithmic Perspective on Imitation Learning (2018, methodology): “[Lemme et al., 2015], it is necessary to establish metrics and benchmarks for comparing existing methods.”
- Learning Partially Contracting Dynamical Systems from Demonstrations (2017, methodology): “The times, the amplitudes, and the directions of the pushes and the changes of goal location were chosen to be the same for all the algorithms and were determined according to the guidelines presented in [35].”
- Geometrical Invariance and Smoothness Maximization for Task-Space Movement Generation (2016, methodology): “In Section VI, we discuss our MG algorithm and the potential of defining movement complexity through optimization criteria, such as the minimum jerk model [3], and based on geometrical invariance (which we introduced in the context of motor control in [2] and [25]).”