Reinforcement Learning Via Practice and Critique Advice

By Kshitij Judah, Saikat Roy, Alan Fern, Thomas G. Dietterich. 2010

86 citations recorded by the source corpus.

We consider the problem of incorporating end-user advice into reinforcement learning (RL). In our setting, the learner alternates between practicing, where learning is based on actual world experience, and end-user critique sessions where advice is gathered. During each critique session the end-user is allowed to analyze a trajectory of the current policy and then label an arbitrary subset of the available actions as good or bad. Our main contribution is an approach for integrating all of the information gathered during practice and critiques in order to effectively optimize a parametric policy. The approach optimizes a loss function that linearly combines losses measured against the world experience and the critique data. We evaluate our approach using a prototype system for teaching tactical battle behavior in a real-time strategy game engine. Results are given for a significant evaluation involving ten end-users showing the promise of this approach and also highlighting challenges involved in inserting end-users into the RL loop.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, neutral): “(2000), Kakade (2001), Schulman et al.”
A comprehensive survey on safe reinforcement learning (2015, background): “On the other hand, Judah et al. (2010) use a teacher that is allowed to observe the execution of the agent's current policy and then scroll back and forth along the trajectory and mark any of the available actions in any state as good or bad.”
Imitation Learning with Demonstrations and Shaping Rewards (2014, background): “Prior work (Judah et al. 2010) studied how to incorporate an expert advice on actions into an RL agent.”
Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds (2017, background): “Thus, many machine learning researchers have taken an interest in using machine learning to train AI agents to play video games (Mnih et There are studies focusing on combining the reward learning methods with human input, such as (Judah et al. 2010; Grifﬁth et al. 2013).”
Reinforcement learning from simultaneous human and MDP reward (2012, background): “consider a learning scenario that alternates between “practice”, where actual world experience is gathered, and an offline labeling of actions as good or bad by a human critic [6].”
Active Reward Learning from Critiques (2018, background): “[11] enabled an agent to learn a parameterized policy from expert’s critiques by encoding critiques into reward values.”
Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning (2015, unknown): “Other work has also shown that feedback can be combined with reward from some underlying Markov decision process, or some predeﬁned shaping reward [12,13].”
A Review of Real-Time Strategy Game AI (2014, result): “However, slightly better overall results were achieved using human critique only to train the agent, possibly due to humans giving better feedback when they can see an immediate result (Judah et al. 2010).”
Learning Rewards from Linguistic Feedback (2020, background): “In RL, it includes labeling sets of actions as good or bad (Judah et al. 2010; Chris-tiano et al. 2017), learning from demonstrations (Ross and Bagnell 2010; Abbeel and Ng 2004; Ho et al. 2016), and corrections to dialogue agents (Li et al. 2016; Chen et al. 2017).”
Agent-Agnostic Human-in-the-Loop Reinforcement Learning (2017, background): “The agent is not specialized to the protocol, so it is unable to ask the human informative questions as in [4], or will not have an observation model that faithfully represents the process the human uses to generate advice, as in [17, 21].”
Learning From Stories: Using Crowdsourced Narratives to Train Virtual Agents (2021, background): “These systems use human reward signals to shape agent behavior (Knox and Stone 2009; Judah et al. 2010; Loftin et al. 2014).”
Teaching Machines to Describe Images via Natural Language Feedback (2017, background): “[6] iterates between “practice”, during which the agent interacts with the real environment, and a critique session where a human labels any subset of the chosen actions as good or bad.”
A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation (2018, background): “For example, Judah et al. (2010) presented a scenario where users interactively label freely chosen good and bad parts of a policy’s trajectory.”