Learning from Physical Human Corrections, One Feature at a Time

By Andrea V. Bajcsy, Dylan P. Losey, Marcia Kilchenman O'Malley, Anca Dragan. 2018

112 citations recorded by the source corpus.

We focus on learning robot objective functions from human guidance: specifically, from physical corrections provided by the person while the robot is acting. Objective functions are typically parametrized in terms of features, which capture aspects of the task that might be important. When the person intervenes to correct the robot’s behavior, the robot should update its understanding of which features matter, how much, and in what way. Unfortunately, real users do not provide optimal corrections that isolate exactly what the robot was doing wrong. Thus, when receiving a correction, it is difficult for the robot to determine which features the person meant to correct, and which features were changed unintentionally. In this paper, we propose to improve the efficiency of robot learning during physical interactions by reducing unintended learning. Our approach allows the human-robot team to focus on learning one feature at a time, unlike state-of-the-art techniques that update all features at once. We derive an online method for identifying the single feature which the human is trying to change during physical interaction, and experimentally compare this one-at-a-time approach to the all-at-once baseline in a user study. Our results suggest that users teaching one-at-a-time perform better, especially in tasks that require changing multiple features.

Subfields: Artificial Intelligence, Control and Systems Engineering, Social Psychology, Reinforcement Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Social Psychology

In Reinforcement Learning

In Human-Computer Interaction

Citations in the corpus

Learning from Interventions: Human-robot interaction as both explicit and implicit feedback (2020, background): “Similar ideas have been successfully applied to reduce effort in learning from kinesthetic corrections in the manipulator control domain [23, 24, 25] or by monitoring a buffer of future robot actions in social robotic teaching [26].”
Correcting Robot Plans with Natural Language Feedback (2022, background): “Our use of language contrasts with previous work where corrective input of robot behavior came from joystick control [36, 33], kinesthetic feedback [27, 19, 6], or spatial labelling of constraints [44, 9].”
Interactive Language: Talking to Robots in Real Time (2022, background): “Importantly, due to short-horizon skill competency, this shows that language can relax the assumption of undivided operator attention, which is common for prior ways of correcting online robot behavior [32], [34], [56].”
Feature Expansive Reward Learning: Rethinking Human Input (2020, methodology): “ull demonstration, work on online learning from demonstrations (Sec. 3.2 in [10]) has derived the same update with ˝ Hnow the human demonstration. In our implementation, we use corrections and follow [3], which shows that people more easily correct on feature at a time, and only update the index cor”
Learning latent actions to control assistive robots (2021, unknown): “Carefully orchestrating complex movements of high-dimensional robots is difficult for humans, especially when users are limited to a lowdimensional control interface [5].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “In robotics, correction-based feedback has been used for improving policies [Li et al., 2021, Losey et al., 2022, Bajcsy et al., 2018] and plans [Sharma et al., 2022].”