Sim2Real Viewpoint Invariant Visual Servoing by Recurrent Control

By Fereshteh Sadeghi, Alexander Toshev, Eric Jang, Sergey Levine. 2018

76 citations recorded by the source corpus.

Humans are remarkably proficient at controlling their limbs and tools from a wide range of viewpoints. In robotics, this ability is referred to as visual servoing: moving a tool or end-point to a desired location using primarily visual feedback. In this paper, we propose learning viewpoint invariant visual servoing skills in a robot manipulation task. We train a deep recurrent controller that can automatically determine which actions move the end-effector of a robotic arm to a desired object. This problem is fundamentally ambiguous: under severe variation in viewpoint, it may be impossible to determine the actions in a single feedforward operation. Instead, our visual servoing approach uses its memory of past movements to understand how the actions affect the robot motion from the current viewpoint, correcting mistakes and gradually moving closer to the target. This ability is in stark contrast to previous visual servoing methods, which assume known dynamics or require a calibration phase. We learn our recurrent controller using simulated data, synthetic demonstrations and reinforcement learning. We then describe how the resulting model can be transferred to a real-world robot by disentangling perception from control and only adapting the visual layers. The adapted model can servo to previously unseen objects from novel viewpoints on a real-world Kuka IIWA robotic arm. For supplementary videos, see: https://www.youtube.com/watch?v=oLgM2Bnb7fo

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Media Technology, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Media Technology

In Robotics & Control

Citations in the corpus

Driving Policy Transfer via Modularity and Abstraction (2018, background): “n [48,16,51,4,36] and specialized architectures [41,53]. Working with depth maps instead of color images is known to simplify transfer [50,28,27]. When dealing with color images, domain randomization [46,21,42,43] enables direct sim-to-real generalization by maximizing the diversity of textures and ”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “A number of works have studied learning representations and policies that transfer across domains, including transfer from simulation to the real world [1, 33, 34], transfer across different dynamics [35, 36, 37, 2], transfer across robot morphologies with invariant feature spaces [38] and modularit”
Simitate: A Hybrid Imitation Learning Benchmark (2019, neutral): “More recently, interesting Real2Sim [29] approaches train robot controllers entirely in simulation and are successfully transferred to real robots.”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “In robotics, a number of recent works have examined using randomized simulated environments [61, 25, 38, 3, 24, 52] specifically for simulation-to-real world transfer for grasping and other similar manipulation tasks, extending on prior work on randomization for collision-free robotic indoor flight ”
Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision (2020, methodology): “To facilitate the transition, Sim2Real learning methods can be used [230].”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “…et al., 2018; Lange et al., 2012; OpenAI et al., 2018; Schenck & Fox, 2016; Matas et al., 2018; James et al., 2017; Singh et al., 2019), including in goal-conditioned settings (Kaelbling, 1993; Schaul et al., 2015; Andrychowicz et al., 2017; Sadeghi et al., 2018; Sadeghi, 2019; Nair et al., 2018a).”