Learning Visually Guided Latent Actions for Assistive Teleoperation

By Siddharth Karamcheti, A. Zhai, Dylan P. Losey, Dorsa Sadigh. 2021

28 citations recorded by the source corpus.

It is challenging for humans -- particularly those living with physical disabilities -- to control high-dimensional, dexterous robots. Prior work explores learning embedding functions that map a human's low-dimensional inputs (e.g., via a joystick) to complex, high-dimensional robot actions for assistive teleoperation; however, a central problem is that there are many more high-dimensional actions than available low-dimensional inputs. To extract the correct action and maximally assist their human controller, robots must reason over their context: for example, pressing a joystick down when interacting with a coffee cup indicates a different action than when interacting with knife. In this work, we develop assistive robots that condition their latent embeddings on visual inputs. We explore a spectrum of visual encoders and show that incorporating object detectors pretrained on small amounts of cheap, easy-to-collect structured data enables i) accurately and robustly recognizing the current context and ii) generalizing control embeddings to new objects and tasks. In user studies with a high-dimensional physical robot arm, participants leverage this approach to perform new tasks with unseen objects. Our results indicate that structured visual representations improve few-shot performance and are subjectively preferred by users.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Control and Systems Engineering, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In LLM Agents & Tool Use

Citations in the corpus

LILA: Language-Informed Latent Actions (2021, methodology): “We implement the encoder and decoder as multi-layer feed-forward networks, with the ReLU activation as in prior work [7, 28].”
Learning latent actions to control assistive robots (2021, unknown): “Although not covered in this paper, it is also possible to condition latent actions directly on the robot’s perception, so that s becomes the visual inputs [28].”