Graph-Structured Visual Imitation
By Maximilian Sieb, Xian Zhou, Audrey Huang, Oliver Kroemer, Katerina Fragkiadaki. 2019
79 citations recorded by the source corpus.
We cast visual imitation as a visual correspondence problem. Our robotic agent is rewarded when its actions result in better matching of relative spatial configurations for corresponding visual entities detected in its workspace and teacher's demonstration. We build upon recent advances in Computer Vision,such as human finger keypoint detectors, object detectors trained on-the-fly with synthetic augmentations, and point detectors supervised by viewpoint changes and learn multiple visual entity detectors for each demonstration without human annotations or robot interactions. We empirically show the proposed factorized visual representations of entities and their spatial arrangements drive successful imitation of a variety of manipulation skills within minutes, using a single demonstration and without any environment instrumentation. It is robust to background clutter and can effectively generalize across environment variations between demonstrator and imitator, greatly outperforming unstructured non-factorized full-frame CNN encodings of previous works.
Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Li Fei-Fei
- Luc Van Gool
- Trevor Darrell
- Jitendra Malik
- Yuke Zhu
- Ming-Hsuan Yang
- Dacheng Tao
- Xiaogang Wang
In Control and Systems Engineering
- Li Fei-Fei
- Yuke Zhu
- Danfei Xu
- Peter R. Florence
- Lucas Manuelli
- Linxi (Jim) Fan
- Anima Anandkumar
- Russ Tedrake
Citations in the corpus
- Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “Relevant concurrent works include [19] which proposes autoencoder-style visual training but with a reference image and novel architecture, and [20] which proposes a graphbased reward function using a fixed set of correspondences.”
- MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, background): “To reduce the domain gap, another thread of work [2, 27, 28, 47, 52, 53, 58] utilizes indomain human videos, where human directly interacts with the robot task environment with their own hands.”
- DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “Another line of work in imitation learning is to train policies to follow the expert demonstrations [43, 52, 49, 71, 68, 72, 74, 21, 51, 73].”
- VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “Unlike prior work (Florence et al., 2019; Sieb et al., 2019; Zhu et al., 2022), VIMA adopts an object-centric representation that computes tokens from bounding box coordinates and cropped RGB patches.”
- Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “This has motivated a variety of graph-based scene representations [1, 19, 28, 29, 39] developed in the robotics and computer vision community.”