Learning preferences for manipulation tasks from online coactive feedback

By Ashesh Jain, Shikhar Sharma, Thorsten Joachims, Ashutosh Saxena. 2015

132 citations recorded by the source corpus.

We consider the problem of learning preferences over trajectories for mobile manipulators such as personal robots and assembly line robots. The preferences we learn are more intricate than simple geometric constraints on trajectories; they are rather governed by the surrounding context of various objects and human interactions in the environment. We propose a coactive online learning framework for teaching preferences in contextually rich environments. The key novelty of our approach lies in the type of feedback expected from the user: the human user does not need to demonstrate optimal trajectories as training data, but merely needs to iteratively provide trajectories that slightly improve over the trajectory currently proposed by the system. We argue that this coactive preference feedback can be more easily elicited than demonstrations of optimal trajectories. Nevertheless, theoretical regret bounds of our algorithm match the asymptotic rates of optimal trajectory algorithms. We implement our algorithm on two high-degree-of-freedom robots, PR2 and Baxter, and present three intuitive mechanisms for providing such incremental feedback. In our experimental evaluation we consider two context rich settings, household chores and grocery store checkout, and show that users are able to train the robot with just a few feedbacks (taking only a few minutes).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning from Physical Human Corrections, One Feature at a Time (2018, background): “Under the ideal objective function parameters, the corrected behavior has to have a lower cost than the robot’s current behavior [3, 11].”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “Although some methods incorporate multiple instruction types Jain et al. [2015], this research direction has not been well-investigated yet.”
Inverse Reward Design (2017, background): “, 2015), preference-based reward learning uses answers to comparison queries (Jain et al., 2015), and even what the human wants (HadfieldMenell et al.”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, methodology): “Prior work [10] has also modeled a variant of corrections in which the human provides an improved trajectory ξimproved which is treated as better than the robot’s original ξR.”
Active Preference-Based Learning of Reward Functions (2017, background): “Typically in preference-based learning, the queries are actively selected by searching some discrete or sampled set (e.g., [3, 9, 12, 13, 20, 21]).”
Robobarista: Object Part Based Transfer of Manipulation Trajectories from Crowd-Sourcing in 3D Pointclouds (2015, background): “Among previous efforts to scale teaching to the crowd [9, 52, 21], Forbes”
Preference-based Reinforcement Learning with Finite-Time Guarantees (2020, background): “Preference-based Reinforcement Learning (PbRL, [28]) is a framework to solve RL using preferences, and has been widely applied in multiple areas including robot teaching [20, 19, 11], game playing [29, 31], and in clinical trials [36].”
Correcting Robot Plans with Natural Language Feedback (2022, result): “Our use of language contrasts with previous work where corrective input of robot behavior came from joystick control [36, 33], kinesthetic feedback [27, 19, 6], or spatial labelling of constraints [44, 9].”
Learning Robot Objectives from Physical Human Interaction (2017, methodology): “We introduce a solution that adapts learning from demonstration approaches to our online pHRI setting [4, 5], but derive it as an approximate solution to the problem above.”
Feature Expansive Reward Learning: Rethinking Human Input (2020, methodology): “Copyright 2020 by the author(s). erence for how to complete the task, enabling the robot to update its reward function online (Bajcsy et al., 2017; Jain et al., 2015; Bajcsy et al., 2018).”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “…(Wirth et al., 2016; Wilson et al., 2012; Akrour et al., 2011, 2012; Jain et al., 2013, 2015; Gritsenko and Berenson, 2014; Zucker et al., 2010; Jain et al., 2013, 2015; Gritsenko and Berenson, 2014) also require the definition of a feature-based description of a trajectory, which we denote as…”