Comparing Action-Query Strategies in Semi-Autonomous Agents

By Robert S. Cohn, Edmund H. Durfee, Satinder Singh. 2011

42 citations recorded by the source corpus.

We consider settings in which a semi-autonomous agent has uncertain knowledge about its environment, but can ask what action the human operator would prefer taking in the current or in a potential future state. Asking queries can improve behavior, but if queries come at a cost (e.g., due to limited operator attention), the value of each query should be maximized. We compare two strategies for selecting action queries: 1) based on myopically maximizing expected gain in long-term value, and 2) based on myopically minimizing uncertainty in the agent's policy representation. We show empirically that the first strategy tends to select more valuable queries, and that a hybrid method can outperform either method alone in settings with limited computation.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “Recent work on active learning for inverse RL has sought to reduce the required number of demonstrations [2, 6, 26, 5], but”
Learning with Humans in the Loop (2020, background): “The closest prior work is in sliding autonomy [16, 34, 10, 56] and active learning [15], where the robot can request user intervention in challenging or uncertain situations.”
Active Imitation Learning via Reduction to I.I.D. Active Learning (2012, background): “There has been limited work on active IRL (Lopes, Melo, and Montesano 2009; Cohn, Durfee, and Singh 2011), where the uncertainty about the reward function is used to select the “most informative” query.”
Active Reward Learning from Critiques (2018, background): “[9] proposed to use myopic expected value of information as the measure for selecting action queries instead of policy entropy.”
Teacher-Student Framework: a Reinforcement Learning Approach (2014, background): “[6], advice requested by the student, noisy advice).”
Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications (2018, background): “There is a large body of work on using active learning for IRL (Lopes, Melo, and Montesano 2009; Cohn, Durfee, and Singh 2011; Cui and Niekum 2017; Sadigh et al. 2017; Brown, Cui, and Niekum 2018).”
Algorithmic and Human Teaching of Sequential Decision Tasks (2012, background): “…such as preference elicitation from users (Fürnkranz and Hüllermeier 2010; Viappiani and Boutilier 2010) or interactive learning agents (Chernova and Veloso 2007; Grollman and Jenkins 2007), with extensions to the IRL framework (Lopes, Melo, and Montesano 2009; Cohn, Durfee, and Singh 2011).”
Imitation Learning with Demonstrations and Shaping Rewards (2014, methodology): “We use a recent implementation (Cohn, Durfee, and Singh 2011), where there are three traffic lanes and two shoulder lanes.”
Risk-Aware Active Inverse Reinforcement Learning (2018, methodology): “To address this issue, active IRL algorithms [5, 6, 7] have been proposed that reason about uncertainty and information gain to select queries that are expected to be the most informative under certain criteria.”