Learning Trajectory Preferences for Manipulators via Iterative Improvement

By Ashesh Jain, Brian Wojcik, Thorsten Joachims, Ashutosh Saxena. 2013

141 citations recorded by the source corpus.

We consider the problem of learning good trajectories for manipulation tasks. This is challenging because the criterion defining a good trajectory varies with users, tasks and environments. In this paper, we propose a co-active online learning framework for teaching robots the preferences of its users for object manipulation tasks. The key novelty of our approach lies in the type of feedback expected from the user: the human user does not need to demonstrate optimal trajectories as training data, but merely needs to iteratively provide trajectories that slightly improve over the trajectory currently proposed by the system. We argue that this co-active preference feedback can be more easily elicited from the user than demonstrations of optimal trajectories, which are often challenging and non-intuitive to provide on high degrees of freedom manipulators. Nevertheless, theoretical regret bounds of our algorithm match the asymptotic rates of optimal trajectory algorithms. We demonstrate the generalizability of our algorithm on a variety of grocery checkout tasks, for whom, the preferences were not only influenced by the object being manipulated but also by the surrounding environment.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve ﬁrst explicitly estimating the latent scoring function (i.e. the reward model) and subsequently optimizing it [16, 9, 12, 34, 19].”
Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, methodology): “This framework widely applies to multiple areas including robot training, stock-prediction, recommender systems, clinical trials, etc. (Novoseller et al., 2019; Sadigh et al., 2017; Christiano et al., 2017; Kupcsik et al., 2018; Jain et al., 2013; Wirth et al., 2017).”
Learning Dynamic Robot-to-Human Object Handover from Human Feedback (2016, background): “Jain et al. learn manipulation trajectories from human preferences [17].”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “Data-driven approaches for reward specification [29, 1, 11, 17, 12, 48, 33, 9, 27, 4, 18] seek to overcome this issue, but typically require demonstration data to acquire rewards.”
Active Reward Learning (2014, background): “[18] have proposed the iterative improvement of trajectories.”
Learning Awareness Models (2018, background): “…and other creatures (Heess et al., 2017), or the real world in the form of virtual assistants, self driving vehicles (Bojarski et al., 2016), and household robots (Jain et al., 2013); we are increasingly faced with the need to build systems that understand and reason about the world around them.”
Interactive machine learning for health informatics: when do we need the human-in-the-loop? (2016, background): “Jain et al. [71] considered the problem of learning good trajectories for manipulation tasks, which is challenging because the criteria deﬁning good trajectories varies with users, tasks, and environments.”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, background): “…systems, clinical trials, large language models etc. (Novoseller et al., 2019; Sadigh et al., 2017; Christiano et al., 2017b; Kupcsik et al., 2018; Jain et al., 2013; Wirth et al., 2017; Knox and Stone, 2008; MacGlashan et al., 2017; Christiano et al., 2017a; Warnell et al., 2018; Brown et al.,…”
A Survey of Preference-Based Reinforcement Learning Methods (2017, methodology): “…(Wirth et al., 2016; Wilson et al., 2012; Akrour et al., 2011, 2012; Jain et al., 2013, 2015; Gritsenko and Berenson, 2014; Zucker et al., 2010; Jain et al., 2013, 2015; Gritsenko and Berenson, 2014) also require the definition of a feature-based description of a trajectory, which we denote as…”
Coactive Learning (2015, methodology): “We formalize Coactive Learning as a general model of interaction between a learning system and its user, define a suitable notion of regret, and validate the key modeling assumption – namely whether observable user behavior can provide valid feedback in our model – in a user study for web search.”
Model Alignment as Prospect Theoretic Optimization (2024, background): “This single stage of optimization distinguishes DPO from the conventional approach in preference-based RL, which learns a reward and then fits the policy to those rewards (Jain et al., 2013; Busa-Fekete et al., 2014).”
Programming by Feedback (2014, background): “Another scheme called co-active policy learning (CPL) is presented by Jain et al. (2013).”
Preference-based Reinforcement Learning with Finite-Time Guarantees (2020, background): “Preference-based Reinforcement Learning (PbRL, [28]) is a framework to solve RL using preferences, and has been widely applied in multiple areas including robot teaching [20, 19, 11], game playing [29, 31], and in clinical trials [36].”
Learning from Interventions: Human-robot interaction as both explicit and implicit feedback (2020, background): “Similar ideas have been successfully applied to reduce effort in learning from kinesthetic corrections in the manipulator control domain [23, 24, 25] or by monitoring a buffer of future robot actions in social robotic teaching [26].”
Active reward learning with a novel acquisition function (2015, background): “In a combination of IRL and preference learning, Jain et al. (2013) have proposed the iterative improvement of trajectories.”