Deep Reinforcement Learning from Policy-Dependent Human Feedback

By Dilip Arumugam, Jun Ki Lee, Sophie Saskin, Michael L. Littman. 2019

111 citations recorded by the source corpus.

To widen their accessibility and increase their utility, intelligent agents must be able to learn complex behaviors as specified by (non-expert) human users. Moreover, they will need to learn these behaviors within a reasonable amount of time while efficiently leveraging the sparse feedback a human trainer is capable of providing. Recent work has shown that human feedback can be characterized as a critique of an agent's current behavior rather than as an alternative reward signal to be maximized, culminating in the COnvergent Actor-Critic by Humans (COACH) algorithm for making direct policy updates based on human feedback. Our work builds on COACH, moving to a setting where the agent's policy is represented by a deep neural network. We employ a series of modifications on top of the original COACH algorithm that are critical for successfully learning behaviors from high-dimensional observations, while also satisfying the constraint of obtaining reduced sample complexity. We demonstrate the effectiveness of our Deep COACH algorithm in the rich 3D world of Minecraft with an agent that learns to complete tasks by mapping from raw pixels to actions using only real-time human feedback in 10-15 minutes of interaction.

Subfields: Artificial Intelligence, Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Alignment & RLHF

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, methodology): “COACH (MacGlashan et al., 2017; Arumugam et al., 2018) is an algorithm that trains a policy from feedback in the form of an advantage function (Sutton & Barto, 2018).”
Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “Representative works include the TAMER framework [15, 45], and the COACH framework [25, 4].”
Learning to Interactively Learn and Assist (2019, background): “Examples of such interactions include allowing user interventions or agent requests, for demonstrations (Kelly et al. 2018), rewards (Warnell et al. 2018; Arumugam et al. 2019), or preferences (Christiano et al. 2017).”
DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback (2018, unknown): “…RL settings: ( BINARY FEEDBACK , [9] Tenorio-Gonzalez et al. 2010 [10] (voice) Pilarski et al. 2011 [11] Grifﬁth et al. 2013 [12] MacGlashan et al. 2017 [13] Arumugam et al. 2018 [14] Warnel et al. 2018 [15] Ours (facial expression) DELAY , STOCHASTICITY , UNSUSTAINABLITY , NATU - RAL REACTION ).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “Attempting to discretize this form of feedback using a Likert scale (a range of discrete ratings; e.g., very bad, bad, ok, good, very good) simplifies the process of feedback collection [Knox and Stone, 2008, MacGlashan et al., 2017, Arumugam et al., 2019].”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, background): “An extension to domains with large state spaces, Deep-COACH, was presented in [2].”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “…{ s j , a j , s j +1 , r int j } Bj =1 ∼ B 10: Optimize L SACcritic in (1) and L SACact in (2) with respect to θ and φ 11: end for 12: end for 13: return B , π φ − E s ∼ p ( s ) [log p ( s )] as an intrinsic reward (Hazan et al., 2019; Lee et al., 2019; Hao & Pieter, 2021; Seo et al., 2021).”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “[53] Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman.”