Self-Supervised Correspondence in Visuomotor Policy Learning

By Peter R. Florence, Lucas Manuelli, Russ Tedrake. 2019

196 citations recorded by the source corpus.

In this letter, we explore using self-supervised correspondence for improving the generalization performance and sample efficiency of visuomotor policy learning. Prior work has primarily used approaches such as autoencoding, pose-based losses, and end-to-end policy optimization in order to train the visual portion of visuomotor policies. We instead propose an approach using self-supervised dense visual correspondence training and show that this enables visuomotor policy learning with surprisingly high generalization performance with modest amounts of data. Using imitation learning, we demonstrate extensive hardware validation on challenging manipulation tasks with as few as 50 demonstrations. Our learned policies can generalize across classes of objects, react to deformable object configurations, and manipulate textureless symmetrical objects in a variety of backgrounds, all with closed-loop, real-time vision-based policies. Simulated imitation learning experiments suggest that correspondence training offers sample complexity and generalization benefits compared to autoencoding and end-to-end training.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “, weights associated with one but not other modalities, frozen for downstream tasks (Zhai et al., 2021; Florence et al., 2019; Tsimpoukelli et al., 2021; Zakka et al., 2022).”
Implicit Behavioral Cloning (2021, methodology): “Perhaps the success of BC comes from its simplicity: it has the lowest data collection 246 requirements (no reward labels or on-policy data required), can be data-efficient [5, 25], and it is arguably 247 the simplest to implement and easiest to tune (with fewer hyperparameters than RL-based methods”
Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, methodology): “• BC-image: A deep imitation learning model adapted from the end-to-end model variant of [9].”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “End-to-end models for vision-based manipulation learn faster when integrated with such object-centric representations [10], providing sample-efficiency benefits but retaining the data-collection and representational constraints of object-centricness.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Alternatively, dense descriptors [27, 28, 29] and keypoint representations [30, 31, 32] forgo segmentation and pose representations, but do not reason about sequential actions and struggle to represent scenes with variable numbers of objects.”
Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning (2020, methodology): “This problem doesn’t arise in the imitation learning setting of [9] which performs supervised learning from zt→a t , and thus can learn to ignore components of the latent state zt that aren’t useful for predicting the action a expert t .”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “Unlike prior work (Florence et al., 2019; Sieb et al., 2019; Zhu et al., 2022), VIMA adopts an object-centric representation that computes tokens from bounding box coordinates and cropped RGB patches.”
S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning (2021, background): “There has also been recent works showing the effectiveness of simple data augmentation in imitation learning [35, 36].”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, methodology): “In contrast, recent IL methods built upon deep neural networks can learn reactive policies from raw demonstration data [20, 2, 3, 21–24].”
A Framework for Efficient Robotic Manipulation (2020, methodology): “Since our method uses demonstrations, we include pull, which has been used in prior work on imitation learning [41], [73].”
TAP-Vid: A Benchmark for Tracking Any Point in a Video (2022, unknown): “Keypoint discovery aims to discover keypoints and correspondence on more general objects, typically with the goal of robotic manipulation [14, 27, 28, 67, 86], but these typically train from large datasets depicting a single object.”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “…methods such as autoencoders (Finn et al. 2016b; Ghadirzadeh et al. 2017), contrastive losses (Sermanet et al. 2018), and correspondence learning (Florence et al. 2018, 2019), provide a reasonable solution in cases where the inductive biases of the unsupervised algorithm effectively match the…”
Real-World Robot Learning with Masked Visual Pre-training (2022, background): “…learning visual autoencoders on trajectories [35]; learning spatiotemporal representations through videos [36, 37]; learning visual correspondence [38]; utilizing non-parametric nearest-neighbor retrieval [39]; and conducting visual self-supervised learning on pre-collected demonstrations [40].”
Behavior Transformers: Cloning k modes with one stone (2022, background): “Behavior cloning is a form of imitation learning that tries to model the action of the expert given the observation which is often used in real-world applications [84, 85, 84, 69, 24, 83].”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “The simplest form of explicit policies maps from world state or observation directly to action (Pomerleau, 1988; Zhang et al., 2018; Florence et al., 2020; Ross et al., 2011; Toyer et al., 2020; Rahmatizadeh et al., 2018); Bojarski et al., 2016).”