Policy Shaping: Integrating Human Feedback with Reinforcement Learning

By Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles Lee Isbell Jr., Andrea Lockerd Thomaz. 2013

484 citations recorded by the source corpus.

Subfields: Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Alignment & RLHF

In Reinforcement Learning

Citations in the corpus

The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, background): “Anecdotally, some powerful recent models are trained using scrapes of much of the Internet, then further fine-tuned with reinforcement learning from human feedback (RLHF) [Griffith et al., 2013, OpenAI, 2023].”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…1998; Ng & Russell, 2000; Abbeel & Ng, 2004; Argall et al., 2009); providing feedback in the form of scores (El Asri et al., 2016), actions (Grifﬁth et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), or preferences over trajectories (Fürnkranz et al., 2012;…”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, background): “Here human feedback is direct policy advice as in the Advice framework [11], indicating whether an action is preferable regardless of the agent’s current policy.”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “[Grifﬁth et al. , 2013] assumes such feed-back could be inconsistent.”
Interactive Learning from Policy-Dependent Human Feedback (2017, background): “Two other closely related approaches are SABL (Loftin et al., 2015) and Policy Shaping (Grifﬁth et al., 2013).”
Active Reward Learning (2014, methodology): “The Advise framework [14] also uses the term shaping to describe the process of modelling an oracle and are actively requesting labels for an agent’s action (good or bad).”
Consequences of Misaligned AI (2021, background): “This includes systems that learn from direct comparisons between options (9; 11; 13), demonstrations of optimal behavior (26; 1; 36), corrective feedback (7; 15; 2), and proxy metrics (18).”
Deep Reinforcement Learning from Policy-Dependent Human Feedback (2019, unknown): “icy shaping (Griffith et al., 2013) and SABL (Loftin et al.”
Multi-modal integration of dynamic audiovisual patterns for an interactive reinforcement learning scenario (2016, background): “1) in order to speed up the apprenticeship process by either reward or policy shaping [6] [7].”
Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning (2015, methodology): “In contrast to that work, ours focuses on handling different trainer feedback strategies, whereas Policy Shaping assumes actions which do not receive explicit trainer feedback are uninformative as to the trainer’s policy (though still informative about the underlying MDP).”
PERCEPTUAL GOAL SPECIFICATIONS FOR REINFORCEMENT LEARNING (2017, background): “Alternatively, policy shaping takes a more hands-off approach, and provides policy “advice”, as opposed to explicit rewards [20].”
Generative artificial intelligence (2023, unknown): “One approach to steer the quality of outputs from GAI models is via reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Griffith et al., 2013).”
Agent-Agnostic Human-in-the-Loop Reinforcement Learning (2017, background): “elops agent-speciﬁc protocols for human interaction. That is, protocols for human interaction or advice that are designed for a speciﬁc RL algorithm (such as Q-learning). For instance, Grifﬁth et al. [17] investigate the power of policy advice for a Bayesian Q-Learner. Other works assume that the st”
Bridging the Gap: Converting Human Advice into Imagined Examples (2019, background): “Policy shaping instead translates human feedback into direct policy feedback; the humanprovided rewards over the agent’s actions are used to infer the optimal policy rather than the reward function (Griffith et al., 2013).”
Interactively shaping robot behaviour with unlabeled human instructions (2019, methodology): “Overall, policy-shaping methods have been shown to perform better than reward shaping because they do not interfere with the reward function, hence they avoid convergence problems [12, 20].”