Active Preference-Based Learning of Reward Functions

By Dorsa Sadigh, Anca Dragan, S. Sastry, Sanjit A. Seshia. 2017

477 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Reinforcement Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Signal Processing

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve ﬁrst explicitly estimating the latent scoring function (i.e. the reward model) and subsequently optimizing it [16, 9, 12, 34, 19].”
Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “This framework widely applies to multiple areas including robot training, stock-prediction, recommender systems, clinical trials, etc. (Novoseller et al., 2019; Sadigh et al., 2017; Christiano et al., 2017; Kupcsik et al., 2018; Jain et al., 2013; Wirth et al., 2017).”
A Framework for Data-Driven Robotics (2019, background): “Preferences can be extracted with whole episode comparisons [54]–[57] or shorter clip comparisons [58], [59].”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, unknown): “Most previous work, such as Christiano et al. (2017); Sadigh et al. (2017); Bai et al. (2022a); Kim et al. (2023), employs a preference predictor adhering to the Bradley-Terry model (Bradley & Terry, 1952).”
Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences (2020, methodology): “Preferences over demonstrations are intuitive for humans to provide (Sadigh et al., 2017; Christiano et al., 2017; Palan et al., 2019) and allow for better-than-demonstrator performance (Brown et al., 2019a).”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Motivated by this, many works have studied the reward speciﬁcation, including inverse reinforcement learning [38] with robot demonstrations [39, 40, 41, 42, 43], learning rewards from user preferences [44, 45, 46, 47], and learning rewards from videos of humans [48, 49].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Two examples are using active reward learning to determine when to collect feedback and which feedback to request first [Sadigh et al., 2017], and leveraging dialogue models to learn desired feedback-seeking patterns [Krasheninnikov et al., 2022].”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, background): “…fields of game playing, robot training, stock-prediction, recommender systems, clinical trials, large language models etc. (Novoseller et al., 2019; Sadigh et al., 2017; Christiano et al., 2017b; Kupcsik et al., 2018; Jain et al., 2013; Wirth et al., 2017; Knox and Stone, 2008; MacGlashan et al.,…”
Policy Gradient Bayesian Robust Optimization for Imitation Learning (2021, methodology): “Reward functions are often parameterized as a linear combination of known features (Abbeel & Ng, 2004; Ziebart et al., 2008; Sadigh et al., 2017) or as a deep neural network (Ho & Ermon, 2016; Fu et al., 2017).”
Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications (2018, background): “There is a large body of work on using active learning for IRL (Lopes, Melo, and Montesano 2009; Cohn, Durfee, and Singh 2011; Cui and Niekum 2017; Sadigh et al. 2017; Brown, Cui, and Niekum 2018).”
Reward-rational (implicit) choice: A unifying formalism for reward learning (2020, methodology): “…about the intended reward from demonstrations (IRL) [Ng and Russell, 2000, Abbeel and Ng, 2004], by asking us to make comparisons between trajectories [Wirth et al., 2017, Sadigh et al., 2017, Christiano et al., 2017], or by grounding our instructions [MacGlashan et al., 2015, Fu et al., 2019].”
The EMPATHIC Framework for Task Learning from Implicit Human Feedback (2020, background): “[5] D.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, background): “It can be learned explicitly, from demonstrations (Finn et al. 2016a), from human annotation (Cabi et al. 2019), from human preferences (Sadigh et al. 2017; Christiano et al. 2017), or from multiple sources of human feedback (Bıyık et al. 2020).”
Reward Design with Language Models (2023, methodology): “Another method of specifying user objectives is to learn them from expert demonstrations (Ross et al., 2011) or preferences Sadigh et al. (2017).”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “Another exciting direction is to combine our methods with active preference learning [56, 57, 58, 59] to generate open-ended queries that maximally reduce uncertainty about human preferences.”
Uncertainty Estimation for Language Reward Models (2022, background): “More sophisticated approaches have also been developed, but are difﬁcult to scale and have only been demonstrated in low-dimensional environments [35, 1].”
Cognitive Architectures for Language Agents (2023, background): “…language may ask for help (Ren et al., 2023; Nguyen et al., 2022b; 2019; Nguyen and Daumé III, 2019) or clarification (Biyik and Palan, 2019; Sadigh et al., 2017; Padmakumar et al., 2022; Thomason et al., 2020; Narayan-Chen et al., 2019) – or entertain or emotionally help people (Zhang et…”
Planning and Decision-Making for Autonomous Vehicles (2018, background): “By contrast, the weights of the reward function can also be found by having a human driver choose a preferred trajectory iteratively from a set of two candidate trajectories (109).”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “…the supervision is easy to provide yet information-rich (Akrour et al., 2011; Pilarski et al., 2011; Akrour et al., 2012; Wilson et al., 2012; Sugiyama et al., 2012; Wirth & Fürnkranz, 2013; Wirth et al., 2016; Sadigh et al., 2017; Biyik & Sadigh, 2018; Leike et al., 2018; Biyik et al., 2020).”