Shared Autonomy via Hindsight Optimization

By Shervin Javdani, Siddhartha S. Srinivasa, J. Bagnell. 2015

228 citations recorded by the source corpus.

In shared autonomy, user input and robot autonomy are combined to control a robot to achieve a goal. Often, the robot does not know a priori which goal the user wants to achieve, and must both predict the user’s intended goal, and assist in achieving that goal. We formulate the problem of shared autonomy as a Partially Observable Markov Decision Process with uncertainty over the user’s goal. We utilize maximum entropy inverse optimal control to estimate a distribution over the user’s goal based on the history of inputs. Ideally, the robot assists the user by solving for an action which minimizes the expected cost-to-go for the (unknown) goal. As solving the POMDP to select the optimal action is intractable, we use hindsight optimization to approximate the solution. In a user study, we compare our method to a standard predict-then-blend approach. We find that our method enables users to accomplish tasks more quickly while utilizing less input. However, when asked to rate each system, users were mixed in their assessment, citing a tradeoff between maintaining control authority and accomplishing tasks quickly.

Subfields: Artificial Intelligence, Automotive Engineering, Social Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Social Psychology

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “, 2013), brain-computer interfaces (Javdani et al., 2015), and forecasting of human behavior (Huang and Kitani, 2014; Huang et al.”
End-to-End Driving Via Conditional Imitation Learning (2017, background): “[15] studied a scenario where a robot assists a human and changes its behavior depending on its estimate of the human’s goal.”
Inverse Reward Design (2017, background): “…reinforcement learning uses human demonstrations (Ng & Russell, 2000; Ziebart et al., 2008), shared autonomy uses human operator control signals (Javdani et al., 2015), preference-based reward learning uses answers to comparison queries (Jain et al., 2015), and even what the human wants…”
Autonomy in Rehabilitation Robotics: An Intersection (2018, unknown): “Intent inference often plays a role in this control allocation, and research explores improving (136, 142) and even influencing (143) this inference.”
Shared Autonomy via Deep Reinforcement Learning (2018, unknown): “We can instantiate Bayesian goal inference by using maximum entropy inverse reinforcement learning [34] with a goal-parameterized Q function trained via Q-learning separately from our agent, analogously to prior work [13].”
Learning Robot Objectives from Physical Human Interaction (2017, methodology): “While IRL can learn the robot’s objective function after one or more demonstrations of a task, online inference is possible when the objective is simply to reach a goal state, and the robot moves through free space [12, 13, 14].”
Human-in-the-Loop Optimization of Shared Autonomy in Assistive Robotics (2017, unknown): “In particular, there are instances of assistance resulting in higher user dissatisfaction [11], and users preferring to be in control and more cautious [13].”
Shared autonomy via hindsight optimization for teleoperation and teaming (2017, methodology): “† In our prior work (Javdani et al., 2015), we used C r κ ( x,a,u ) = C u κ ( x,a ) + ( a − u ) 2 in a diﬀerent framework where only the robot action transitions the state.”
Learning with Humans in the Loop (2020, methodology): “ON THE UTILITY OF MODEL LEARNING IN HUMAN-ROBOT INTERACTION 8 reinforcement learning, and the ToM-based method will plan the robot’s actions using the learned model.”
Symbiotic human-robot collaborative assembly (2019, background): “Extending the applicability to a higher number of human possible tasks and to industrial robots, Pellegrinelli et al. [195] introduced a framework for human-robot workspace sharing (unsynchronised tasks) based on hindsight optimisation [92].”
On the Utility of Learning about Humans for Human-AI Coordination (2019, background): “Much work focuses on achieving collaboration by planning and learning with (non-optimal) models of human behavior [26, 21, 31], as well as on specific properties of robot behavior that aid collaboration [2, 14, 9].”
Probabilistic Human Intent Recognition for Shared Autonomy in Assistive Robotics (2019, methodology): “The first observation scheme considers a single modality, the proximity to a goal, as this feature is utilized most in existing shared autonomy work [9, 13, 21, 30].”
HARMONIC: A multimodal dataset of assistive human–robot collaboration (2018, methodology): “Finally, for the original robot control code, follow this Prepared using sagej.cls link to a fork of the publicly available implementation of the shared autonomy code we used: https: //github.com/HARPLab/ada_meal_scenario (Javdani et al. 2015).”
Information gathering actions over human internal state (2016, background): “(4) Optimizing expected reward now entails reasoning about the effects that the robot actions will have on what observations the robot will get, i.e., the actions that the human will take in response, and how useful these observations will be in shattering ambiguity about ϕ.”
Should Robots be Obedient? (2017, result): “Like our work, [Javdani et al., 2015] formalize assistive teleoperation as a POMDP in which H’s goals are unknown, and try to optimize an inference of H’s goal.”
Residual Policy Learning for Shared Autonomy (2020, methodology): “These methods [1, 11, 16, 23, 25, 29, 31, 34, 36, 39, 52] exploit the user’s actions and environment observations to infer the user’s goal.”