Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image

By Danny Driess, Jung-Su Ha, Marc Toussaint. 2020

116 citations recorded by the source corpus.

In this paper, we propose a deep convolutional recurrent neural network that predicts action sequences for task and motion planning (TAMP) from an initial scene image. Typical TAMP problems are formalized by combining reasoning on a symbolic, discrete level (e.g. first-order logic) with continuous motion planning such as nonlinear trajectory optimization. Due to the great combinatorial complexity of possible discrete action sequences, a large number of optimization/motion planning problems have to be solved to find a solution, which limits the scalability of these approaches. To circumvent this combinatorial complexity, we develop a neural network which, based on an initial image of the scene, directly predicts promising discrete action sequences such that ideally only one motion planning problem has to be solved to find a solution to the overall TAMP problem. A key aspect is that our method generalizes to scenes with many and varying number of objects, although being trained on only two objects at a time. This is possible by encoding the objects of the scene in images as input to the neural network, instead of a fixed feature vector. Results show runtime improvements of several magnitudes. Video: this https URL

Subfields: Computer Vision and Pattern Recognition, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Robotics & Control

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “We utilize the planner from Driess et al. (2020) to generate the dataset for the planning tasks.”
Physically Embedded Planning Problems: New Challenges for Reinforcement Learning (2020, methodology): “Although recent approaches to task and motion planning have incorporated methods from deep learning and RL (e.g., Chitnis et al., 2016; Driess et al., 2020; Kim and Shimanuki, 2019; Kim et al., 2019), research on task and motion planning has largely occurred independently of developments in RL, and ”
Text2Motion: from natural language instructions to feasible plans (2023, background): “…[19, 20] explore the integration of LLMs with PDDL [1], and [21, 22] focuses on task-level replanning with LLMs. Tangential works shift the representation of plans from action sequences to code [5, 23–25] and embed task queries, robot actions, solution samples, and fallback behaviors as…”
Inventing Relational State and Action Abstractions for Effective and Efficient Bilevel Planning (2022, background): “Other efforts in this space include heuristic learning [70, 71] (sometimes for TAMP [72, 73, 74]), learning to estimate the feasibility of abstract plans [75, 76], and learning to generate reductions of planning models [77, 78].”
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, background): “Alternatively, recent work has explored learning techniques to evaluate the feasibility of plans from visual observations [6, 37].”