Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids

By Yunzhu Li, Jiajun Wu, Russ Tedrake, Josh Tenenbaum, Antonio Torralba. 2018

479 citations recorded by the source corpus.

Real-life control tasks involve matters of various substances---rigid or soft bodies, liquid, gas---each with distinct physical behaviors. This poses challenges to traditional rigid-body physics engines. Particle-based simulators have been developed to model the dynamics of these complex scenes; however, relying on approximation techniques, their simulation often deviates from real-world physics, especially in the long term. In this paper, we propose to learn a particle-based simulator for complex control tasks. Combining learning with particle-based systems brings in two major benefits: first, the learned simulator, just like other particle-based systems, acts widely on objects of different materials; second, the particle-based representation poses strong inductive bias for learning: particles of the same type have the same dynamics within. This enables the model to quickly adapt to new environments of unknown dynamics within a few observations. We demonstrate robots achieving complex manipulation tasks using the learned simulator, such as manipulating fluids and deformable foam, with experiments both in simulation and in the real world. Our study helps lay the foundation for robot learning of dynamic scenes with particle-based representations.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistical and Nonlinear Physics

Citations in the corpus

Learning to Simulate Complex Physics with Graph Networks (2020, methodology): “Li et al. (2018)’s DPI studied four datasets of fluid, deformable, and solid simulations, and presented four different, distinct architectures, which were similar to SanchezGonzalez et al. (2018)’s, with additional features such as as hierarchical latent nodes.”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, background): “…it often requires both geometry and dynamics of environment to be modeled beforehand, various works have studied structural representations using data-driven methods, such as learning object-centric representation [19–34], particle-based dynamics [35–41], and keypoints or descriptors [3, 4, 42–54].”
Multipole Graph Neural Operator for Parametric Partial Differential Equations (2020, background): “Works on multi-resolution graphs with a U-net like structure have also recently began to emerge and seen success in imaging, classification, and semi-supervised learning [39, 1, 2, 20, 30, 34, 31].”
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models (2023, background): “While the literature is vast, they can be broadly categorized into those that learn the models [90–98] and those that learn the cost/reward or constraints [99– 102, 50, 103], where data are typically collected from in-domain interactions.”
Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction (2020, background): “, modeling physical systems [4,28], drug prediction [31] and social recommendation systems [11].”
Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers (2020, background): “, for rigid-body physics in the context of human reasoning [3, 5, 71] or weather predictions [57], particles are also a popular basis for fluid flow problems [37, 53, 68].”
Discovering Symbolic Models from Deep Learning with Inductive Biases (2020, background): “Recent work has leveraged the inductive biases of Interaction Networks (INs) [6] in their generalized form, the Graph Network , a type of Graph Neural Network [7, 8, 9], to learn models of particle systems in many physical domains [6, 10, 11, 12, 13, 14, 15, 16].”
UvA-DARE (Digital Academic Repository) Neural Relational Inference for Interacting Systems Neural Relational Inference for Interacting Systems (2018, unknown): “, 2017) and predicting physical interactions of particles (Mrowca et al., 2018; Li et al., 2018; Sanchez-Gonzalez et al., 2020).”
Visual Grounding of Learned Physical Models (2020, methodology): “Recently, the DPI-Net (Li et al., 2019a) has achieved strong results in modeling a variety of rigid and deformable object dynamics.”
Modeling, learning, perception, and control methods for deformable object manipulation (2021, methodology): “, force transmission within rigid bodies (5).”
Learning to Control PDEs with Differentiable Physics (2020, methodology): “Lagrangian ﬂuid simulation has been tackled with regression forests (Ladicky et al., 2015), graph neural networks (Mrowca et al., 2018; Li et al., 2019), and continuous convolutions (Ummenhofer et al., 2020).”
Dynamic-Backbone Protein-Ligand Structure Prediction with Multiscale Generative Diffusion Models (2022, methodology): “The architecture of ESDM (Figure1e) is inspired by prior works on 3D graph and attentional neural networks for point clouds [37, 38], rigid-body simulations [39] and biopolymer representation learning [9, 40–42].”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Such human-like physical scene understanding capabilities are also of great importance in practical applications such as industrial robot control [2, 51].”
Learning Physical Graph Representations from Visual Scenes (2020, background): “Recent graph neural network models have shown promise at physical dynamics prediction, but require graph-structured input or supervision [34, 30, 31, 40] – further motivating the unsupervised learning of graph representations from visual input.”