SE3-nets: Learning rigid body motion using deep neural networks

By Arunkumar Byravan, Dieter Fox. 2016

281 citations recorded by the source corpus.

We introduce SE3-Nets which are deep neural networks designed to model and learn rigid body motion from raw point cloud data. Based only on sequences of depth images along with action vectors and point wise data associations, SE3-Nets learn to segment effected object parts and predict their motion resulting from the applied force. Rather than learning point wise flow vectors, SE3-Nets predict SE(3) transformations for different parts of the scene. Using simulated depth data of a table top scene and a robot manipulator, we show that the structure underlying SE3-Nets enables them to generate a far more consistent prediction of object motion than traditional flow based networks. Additional experiments with a depth camera observing a Baxter robot pushing objects on a table show that SE3-Nets also work well on real data.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

In AI for Physical Sciences

Citations in the corpus

Flexible Neural Representation for Physics Prediction (2018, background): “Representations using 3D convolution or point clouds are better at maintaining object shape [46, 47, 10, 36, 37], but do not entirely capture object permanence, and can be computationally inefﬁcient.”
The limits and potentials of deep learning for robotics (2018, background): “…re-composition of component solutions from different places on the spectrum can be achieved with differentiable versions of existing algorithms (one end of the spectrum) that are compatible solutions obtained with back-propagation (other end of the spectrum) [16], [39], [50], [108], [116], [128].”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “This method is related to a range of recently proposed techniques that use predictive models for vision-based control [28, 29, 30, 31, 32].”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “Several algorithms have been proposed for simple, synthetic images [13] and video game environments [14], [15], [16], but have not been evaluated on generalization or in the real world, while other work has also studied model-based RL for individual robotic skills [17], [18], [19].”
Deep visual foresight for planning robot motion (2016, methodology): “[26] learn a model similar to the one that we use here based on prior work [3], except that it predicts 3D rigid motions for constructing future depth images instead of pixel flow transformations for RGB images.”
Self-Supervised Visual Planning with Temporal Skip Connections (2017, methodology): “point clouds obtained from a depth camera [4], requiring point-to-point correspondences over time, which makes it hard to apply to previously unseen objects.”
Stochastic Adversarial Video Prediction (2018, neutral): “An alternative to generating pixels is transforming them by applying a constrained geometric distortion to a previous frame [3,28,29,30,11,31,32,16,13,33,34,35].”
Unsupervised Perceptual Rewards for Imitation Learning (2016, background): “Deep learning has been used in a range of applications within robotic perception and learning [29, 23, 6].”
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight (2019, methodology): “Visual MPC, as well as other video prediction-based planners [9, 16, 35, 45], generally do not succeed at such temporally extended tasks.”
Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control (2019, background): “SE3-Nets (Byravan & Fox, 2017) learn SE (3) transformation of rigid bodies from point cloud data.”
Recent Advances in Robot Learning from Demonstration (2020, methodology): “Supervised approaches, on the other hand, utilize tools such as hierarchical task networks (24) and neural networks (89, 90).”
Interactive Perception: Leveraging Action in Perception and Perception in Action (2016, methodology): “One example is models of multiview or perspective geometry for camera sensors [14], [124].”
PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations (2017, background): “SE3-nets [2] implement assumptions about rigid body transformations.”
P2P-NET: Bidirectional Point Displacement Net for Shape Transform (background): “Recent attempts have been made to learn to transfer surface details [Berkiten et al. 2017], decorative styles [Hu et al. 2017], and to predict piecewise rigid transformations of 3D objects [Byravan and Fox 2016].”
Learning About Objects by Learning to Interact with Them (2020, background): “A large number of past works in the robotics community have explored the problem of segmenting objects by interacting with them [18, 29, 6, 41, 59, 24, 44, 7, 14].”
SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Control (2018, methodology): “For more details, please refer to prior work [4].”
Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation (2017, background): “Other work [34] also considers tasks of pushing objects on a table-top but uses predictive models on point-cloud data and uses a significantly”
Time Reversal as Self-Supervision (2018, background): “These models have been shown to be effective in planning [29], and have even been extended to operate in 3D point cloud space [30], [31].”
Digging Into Self-Supervised Monocular Depth Estimation (2018, methodology): “Inspired by [4], [61] proposed a more sophisticated motion model using multiple motion masks.”