Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control

By Chen Wang, Rui Wang, Danfei Xu, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese. 2021

37 citations recorded by the source corpus.

Imitation Learning (IL) is an effective framework to learn visuomotor skills from offline demonstration data. However, IL methods often fail to generalize to new scene configurations not covered by training data. On the other hand, humans can manipulate objects in varying conditions. Key to such capability is hand-eye coordination, a cognitive ability that enables humans to adaptively direct their movements at task-relevant objects and be invariant to the objects’ absolute spatial location. In this work, we present a learnable action space, Hand-eye Action Networks (HAN) that learns coordinated hand-eye movements from human teleoperated demonstrations. Through a set of challenging multi-stage manipulation tasks, we show that a visuomotor policy equipped with HAN is able to inherit the key spatial invariance property of handeye coordination and achieve generalization to new scene configurations. Additional materials available at https://sites.google.com/stanford.edu/han

Subfields: Cognitive Neuroscience, Control and Systems Engineering, Mechanical Engineering, Representation & Self-Supervised Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Control and Systems Engineering

In Mechanical Engineering

In Representation & Self-Supervised Learning

In Robotics & Control

Citations in the corpus

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023, methodology): “It has been used extensively in prior work for robot manipulation [1,19,25,29–34] — in this work, we use BC to train single-task policies from datasets generated by MimicGen.”