Interactive Policy Learning through Confidence-Based Autonomy

By Sonia Chernova, Manuela M. Veloso. 2014

287 citations recorded by the source corpus.

We present Confidence-Based Autonomy (CBA), an interactive algorithm for policy learning from demonstration. The CBA algorithm consists of two components which take advantage of the complimentary abilities of humans and computer agents. The first component, Confident Execution, enables the agent to identify states in which demonstration is required, to request a demonstration from the human teacher and to learn a policy based on the acquired data. The algorithm selects demonstrations based on a measure of action selection confidence, and our results show that using Confident Execution the agent requires fewer demonstrations to learn the policy than when demonstrations are selected by a human teacher. The second algorithmic component, Corrective Demonstration, enables the teacher to correct any mistakes made by the agent through additional demonstrations in order to improve the policy and future task performance. CBA and its individual components are compared and evaluated in a complex simulated driving domain. The complete CBA algorithm results in the best overall learning performance, successfully reproducing the behavior of the teacher while balancing the tradeoff between number of demonstrations and number of incorrect actions during learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “Nevertheless examples that consider robotic problems already exist for a large variety of problems: building environment maps [67, 118], reinforcement learning [12], body schema learning [25], imitation [61, 27], exploration of objects and body properties [77], manipulation [46], among many others.”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “Prior work has attempted to reduce the number of human annotations needed [10, 23, 30] but relabeling is still required.”
An Algorithmic Perspective on Imitation Learning (2018, background): “Algorithm 2 Confidence-based autonomy algorithm: confident execution and corrective demonstration [Chernova and Veloso, 2009] Input: Demonstration of the action-state pairs D = {(xi, ui)}Ni=1, confidence threshold c0 Initialize the policy π repeat Observe the state x Compute the confidence c(x)…”
Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner (2012, background): “Several works in interactive learning have considered extra reinforcement signals [41], action requests [17, 25] or disambiguation among actions [13].”
Boosted Curriculum Reinforcement Learning (2022, unknown): “…various forms, e.g., via samples (Lazaric et al., 2008; Lazaric & Restelli, 2011), policies (Fern´andez & Veloso, 2006; Fern´andez et al., 2010; Silver et al., 2018), reward functions (Brys et al., 2015) or demonstrations/teachers (Chernova & Veloso, 2009; Billard et al., 2016; Koert et al., 2020).”
Active Learning and Intrinsically Motivated Exploration in Robots : Advances and Challenges (2010, methodology): “Nevertheless, examples that consider robotic problems already exist for a large variety of problems: mapping [21], [22], reinforcement learning [23], body schema learning [25], imitation [26], [27], exploration of objects and body properties [7], manipulation [28], among many others.”
Teleoperation with intelligent and customizable interfaces (2013, methodology): “We envision that the robot can autonomously give the user relevant practice scenarios, in the spirit of work in learning from demonstration on robots actively probing users (Chernova & Veloso, 2009; Cakmak & Thomaz, 2012).”
Abstraction Levels for Robotic Imitation: Overview and Computational Approaches (2010, methodology): “For example, in [31], learning from demonstration is formulated as a classification problem and solved using support-vector machines.”
Active Imitation Learning via Reduction to I.I.D. Active Learning (2012, methodology): “3) We found that CBA is quite sensitive to the threshold adjust- ment mechanism and we were unable to find an alternative mechanism that works across our domains.”
How Humans Teach Agents - A New Experimental Perspective (2012, background): “For example, a more sophisticated agent might have some self-confidence measure and only engage the human when it lacks confidence in making decisions [7].”
HG-DAgger: Interactive Imitation Learning with Human Experts (2018, methodology): “Our approach also differs from [18] in that we use an ensemble of neural network learners instead of Gaussian mixture models.”
Bootstrapping Intrinsically Motivated Learning with Human Demonstrations (unknown): “Many prior systems are strongly dependent on human guidance, unable to learn in the absence of human interaction, such as in some examples of learning by demonstration [3]–[6] or learning by physical guidance [7].”
Learning from Limited Demonstrations (2013, background): “Many of the recent methods focus on incrementally querying the expert in appropriate regions of the state space to improve the learned policy, or to reduce uncertainty [1, 2, 3].”
A comprehensive survey on safe reinforcement learning (2015, methodology): “Finally, although more related to learning from demonstration, Chernova and Veloso (2009) also use the confidence parameter to select between agent autonomy and a request for a demonstration based on the measure of action-selection confidence returned by a classifier.”
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (2010, methodology): “…are used to learn a controller, have proven very useful in practice and have led to state-of-the art performance in a variety of applications (Schaal, 1999; Abbeel and Ng, 2004; Ratliff et al., 2006; Silver et al., 2008; Argall et al., 2009; Chernova and Veloso, 2009; Ross and Bagnell, 2010).”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Some approaches actively ask users for additional (partial) demonstrations in areas of the state space in which confidence is low [59] or risk is high [46], while others rely on a human user to identify when a mistake has been made [281].”