Shared Autonomy via Deep Reinforcement Learning

By Siddharth Reddy, Sergey Levine, Anca Dragan. 2018

222 citations recorded by the source corpus.

In shared autonomy, user input is combined with semi-autonomous control to achieve a common goal. The goal is often unknown ex-ante, so prior work enables agents to infer the goal from user input and assist with the task. Such methods tend to assume some combination of knowledge of the dynamics of the environment, the user's policy given their goal, and the set of possible goals the user might target, which limits their application to real-world scenarios. We propose a deep reinforcement learning framework for model-free shared autonomy that lifts these assumptions. We use human-in-the-loop reinforcement learning with neural network function approximation to learn an end-to-end mapping from environmental observation and user input to agent action, with task reward as the only form of supervision. Controlled studies with users (n = 16) and synthetic pilots playing a video game and flying a real quadrotor demonstrate the ability of our algorithm to assist users with real-time control tasks in which the agent cannot directly access the user's private information through observations, but receives a reward signal and user input that both depend on the user's intent. The agent learns to assist the user without access to this private information, implicitly inferring it from the user's input. This allows the assisted user to complete the task more effectively than the user or an autonomous agent could on their own. This paper is a proof of concept that illustrates the potential for deep reinforcement learning to enable flexible and practical assistive systems.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, background): “While other work focuses on “blending” human inputs with possibly task-agnostic policies within the same action space [23, 24], in this work, we focus on assistive teleoperation, where a user is provided a low-dimensional controller (e.g., a joystick, sip-and-puff device) to directly control a…”
Learning with Humans in the Loop (2020, methodology): “Previous work in this area [30, 19, 48, 55, 9] has focused on some combination of inferring user intent and acting to achieve it.”
Towards Explainable Shared Control using Augmented Reality (2019, background): “One potential reason for this is the growing use of deep learning and other “black-box” approaches to intention estimation [14] and arbitration [15], which are far from explainable to end-users [12].”
Shared Autonomy with Learned Latent Actions (2020, background): “Other works in shared autonomy propose/learn suitable dynamics models to translate user inputs to robot actions [26, 28, 27].”
HARMONIC: A multimodal dataset of assistive human–robot collaboration (2018, background): “Additionally, there has been work in learning and leveraging human policies (using keyboard input) (Reddy et al. 2018b,a) and attention models (using keyboard input and eye gaze) (Zhang et al. 2018) for both assisted and shared robot control in Atari games in an arcade learning environment…”
Controlling Assistive Robots with Learned Latent Actions (2019, background): “Other shared autonomy works consider settings where the human’s input has the same number of DoFs as the robot’s action space [12]–[15].”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “This approach is particularly appealing in domains where human actions are fairly easy to model, such as shared autonomy, where a human policy can be learned by imitating a set of human actors (183, 182).”
Learning Human Objectives by Evaluating Hypothetical Behavior (2019, background): “The idea behind modeling the user’s reward function – as opposed to the user’s policy (Ross et al., 2011), value function (Warnell et al., 2018; Reddy et al., 2018), or advantage function (MacGlashan et al., 2017) – is to acquire a compact, transferable representation of the user’s objectives; not…”
Residual Policy Learning for Shared Autonomy (2020, methodology): “Unlike our framework, [41] require access to the task reward function.”