Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

By Muhammad Asif Rana, Daphne Chen, Seyed Reza Ahmadzadeh, Jacob Williams, Vivian Chu, Sonia Chernova. 2019

18 citations recorded by the source corpus.

We contribute a study benchmarking the performance of multiple motion-based learning from demonstration approaches. Given the number and diversity of existing methods, it is critical that comprehensive empirical studies be performed comparing the relative strengths of these techniques. In particular, we evaluate four approaches based on properties an end user may desire for real-world tasks. To perform this evaluation, we collected data from nine participants, across four manipulation tasks. The resulting demonstrations were used to train 180 task models and evaluated on 720 task reproductions on a physical robot. Our results detail how i) complexity of the task, ii) the expertise of the human demonstrator, and iii) the starting configuration of the robot affect task performance. The collected dataset of demonstrations, robot executions, and evaluations are publicly available. Research insights and guidelines are also provided to guide future research and deployment choices about these approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Evaluation & Benchmarks, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yuke Zhu
2. Li Fei-Fei
3. Silvio Savarese
4. Ajay Mandlekar
5. Roberto Mart'in-Mart'in
6. Danfei Xu
7. Soroush Nasiriany
8. Abhinav Kumar Gupta

In Computer Vision and Pattern Recognition

1. Li Fei-Fei
2. Silvio Savarese
3. Luc Van Gool
4. Jitendra Malik
5. Trevor Darrell
6. Xiaogang Wang
7. Dacheng Tao
8. Abhinav Kumar Gupta

In Control and Systems Engineering

1. Yuke Zhu
2. Ajay Mandlekar
3. Li Fei-Fei
4. Silvio Savarese
5. Danfei Xu
6. Roberto Mart'in-Mart'in
7. Soroush Nasiriany
8. Josiah Wong

In Evaluation & Benchmarks

1. Yuke Zhu
2. Silvio Savarese
3. Li Fei-Fei
4. Ajay Mandlekar
5. Danfei Xu
6. Roberto Mart'in-Mart'in
7. Chen Wang
8. Jitendra Malik

In Robotics & Control

1. Yuke Zhu
2. Ajay Mandlekar
3. Li Fei-Fei
4. Silvio Savarese
5. Roberto Mart'in-Mart'in
6. Danfei Xu
7. Soroush Nasiriany
8. Josiah Wong

Citations in the corpus

• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “[71] also used crowdsourced humans to evaluate learned robot motions with subjective metrics such as safety, while we primarily evaluate our policies using task success rate.”