Differentiable Physics and Stable Modes for Tool-Use and Manipulation Planning

By Marc Toussaint, Kelsey R. Allen, Kevin A. Smith, Josh Tenenbaum. 2018

381 citations recorded by the source corpus.

We propose to formulate physical reasoning and manipulation planning as an optimization problem that integrates first order logic, which we call Logic-Geometric Programming.

Subfields: Biomedical Engineering, Industrial and Manufacturing Engineering, Mechanical Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Industrial and Manufacturing Engineering

In Mechanical Engineering

In Reinforcement Learning

Citations in the corpus

PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “2018) and internal models of physics, shape, and material strength enable advances in tool usage (Toussaint et al. 2018) or construction (Nair, Balloch, and Chernova 2019).”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Classically, this problem has been solved through symbolic planning [59, 61] or optimization [63, 64], but these require explicit primitives and constraints.”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Traditionally, this problem has been solved through optimization [28, 29] or symbolic reasoning [24, 26], but more recently machine learning has been applied to aspects of the problem via learned representations, learned task-primitives, and more [30, 31, 32, 33, 34, 35, 36, 37, 38].”
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “They perform discrete task plans with logic search and continuous motion plans through sampling-based techniques [10, 16] or trajectory optimization [23, 31, 32].”
Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight (2019, methodology): “[44, 8] propose to use logic programming together with known models to algorithmically discover tool-use.”
Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers (2020, methodology): “Our work shares the motivation of previous work to use differentiable components at training time [1, 14, 58, 9] and frameworks for differentiable programming [50, 25, 26, 23].”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “While these approaches have shown promising results on tasks such as simulated tool use [10] and non-vision-based manipulation [9], they rely on the ability to acquire a repertoire of robust, low-level motor skills.”
Propagation Networks for Model-Based Control Under Partial Observation (2018, methodology): “simulators [12], [13] have been deployed for tool manipulation and tool-use planning [14].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “itly take this multi-modal structure into account for planning [391, 171].”
Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image (2020, methodology): “Another approach to TAMP is Logic Geometric Programming (LGP) [42, 43, 44, 45, 18], which combines logic search with trajectory optimization.”
The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving (2019, background): “…research has focused on the role of curiosity and social cues to develop the motor primitives necessary for manipulating simple objects to accomplish a goal (Forestier & Oudeyer, 2016), as well as how to use existing abilities to form more complex plans (Toussaint, Allen, Smith, & Tenenbaum, 2018).”
Modeling Long-horizon Tasks as Sequential Interaction Landscapes (2020, background): “Examples include: focusing on an explicitly modeling geometric constraints as part of the task planning [32], leveraging physics-based heuristics [2, 55], hierarchical planning [24, 36], probabilistic models [31, 17] or integrating action parameters as part of learning forward models [20].”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “For this reason, one appealing alternative is to develop explicit physics-based methods to model and reason about dynamics, as highlighted in the recent development of differentiable physics engines [9, 17, 66, 18, 64] and their applications in robotics [9, 17, 66].”
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids (2018, background): “In robotics, the use of differentiable simulators, together with continuous and symbolic optimization algorithms, has enabled planning for increasingly complex whole body motions with multi-contact and multi-object interactions (Toussaint et al., 2018).”
Learning to Control PDEs with Differentiable Physics (2020, methodology): “Differ-entiable solvers and the adjoint method enable efﬁcient optimization of high-dimensional systems (Toussaint et al., 2018; de Avila Belbute-Peres et al., 2018; Schenck & Fox, 2018).”
Integrated Task and Motion Planning (2020, background): “Sequence First Nilsson∗ (2) Erdem∗ (74, 75) Lagriffoul∗ (65, 66, 67) Pandey∗ (110, 111) Lozano-Pérez∗ (112) Dantam∗ (77, 78, 79) Lo∗ (113) Wolfe∗ (114) Srivastava∗ (76, 60) Garrett∗ (55, 73) Toussaint∗ (61, 68, 69) Shoukry∗ (81, 82, 83) Hadfield-Menell∗ (115)”
Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation (2021, background): “Differentiating through long trajectories is an active area of research[38, 21, 18] and is known to be difficult to optimize[39, 40], thus we defer more advanced differentiable algorithms to future releases.”