SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Planning and Control

By Arunkumar Byravan, Felix Leeb, F. Meier, Dieter Fox. 2017

43 citations recorded by the source corpus.

In this work, we present an approach to deep visuomotor control using structured deep dynamics models. Our deep dynamics model, a variant of SE3-Nets, learns a low-dimensional pose embedding for visuomotor control via an encoder-decoder structure. Unlike prior work, our dynamics model is structured: given an input scene, our network explicitly learns to segment salient parts and predict their pose-embedding along with their motion modeled as a change in the pose space due to the applied actions. We train our model using a pair of point clouds separated by an action and show that given supervision only in the form of point-wise data associations between the frames our network is able to learn a meaningful segmentation of the scene along with consistent poses. We further show that our model can be used for closed-loop control directly in the learned low-dimensional pose space, where the actions are computed by minimizing error in the pose space using gradient-based methods, similar to traditional model-based control. We present results on controlling a Baxter robot from raw depth data in simulation and in the real world and compare against two baseline deep networks. Our method runs in real-time, achieves good prediction of scene dynamics and outperforms the baseline methods on multiple control runs. Video results can be found at: this https URL

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “Learning from interaction and intuitive physics (Agrawal et al. 2016) can also be encoded as priors when exploring the world (Byravan et al. 2018) and internal models of physics, shape, and material strength enable advances in tool usage (Toussaint et al. 2018) or construction (Nair, Balloch, and…”
The limits and potentials of deep learning for robotics (2018, background): “Different variants of deep learning have been shown to successfully learn predictive physics models and robot con-trol policies in a purely data driven way [2], [17], [51], [127].”
Stochastic Adversarial Video Prediction (2018, background): “The success of these methods can also be evaluated indirectly, such as evaluating the learned representations [42,43,44], or using the predictions for planning actions of a robotic system in the real world [45,4,46].”
Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning (2018, methodology): “Unlike [7], we demonstrate complex object manipulations with previously-unseen objects from RGB video.”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, background): “Our work aims to enable a robot to learn a variety of skills without human supervision, hence falling under the category of self-supervised robotic learning [36, 2, 13, 6, 32].”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, background): “Byravan et al. (2017); Hafner et al. (2018); Ebert et al. (2018) have shown that visual model predictive control based on such models can be applied to a variety of different settings.”
Time Reversal as Self-Supervision (2018, background): “These models have been shown to be effective in planning [29], and have even been extended to operate in 3D point cloud space [30], [31].”
Visual Robot Task Planning (2018, background): “A recent line of work in robotics focuses on making structured predictions to inform planning [6, 7, 8, 9, 10]: So far, however, these approaches focus on making relatively short-term predictions, and do”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “Consequently, the ability to foresee and hallucinate the future progression of a scene is a cornerstone of artificial visual understanding with applications including autonomous driving [50, 51, 26], medical treatment [6, 17, 7], and robotic planning [19, 23, 11].”
Planning with Goal-Conditioned Policies (2019, background): “ontext of low-dimensional state spaces [50, 32, 39, 7]. When the observations are high-dimensional, such as images, model errors for direct prediction compound quickly, making model-based RL difﬁcult [15, 13, 5, 14, 26]. Rather than planning directly over image observations, we propose to plan at a ”
Unseen Object Instance Segmentation for Robotic Environments (2020, background): “[50] jointly estimates instance segmentation masks and rigid scene ﬂow, similar to [51], [52].”
KeyIn: Discovering Subgoal Structure with Keyframe-based Video Prediction (2019, background): “Recent work (Byravan et al., 2017; Hafner et al., 2018; Ebert et al., 2018) has shown that visual model predictive control based on such models can be applied to a variety of different settings.”