Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning

By Brian Ichter, Pierre Sermanet, Corey Lynch. 2020

47 citations recorded by the source corpus.

Long-horizon planning in realistic environments requires the ability to reason over sequential tasks in high-dimensional state spaces with complex dynamics. Classical motion planning algorithms, such as rapidly-exploring random trees, are capable of efficiently exploring large state spaces and computing long-horizon, sequential plans. However, these algorithms are generally challenged with complex, stochastic, and high-dimensional state spaces as well as in the presence of narrow passages, which naturally emerge in tasks that interact with the environment. Machine learning offers a promising solution for its ability to learn general policies that can handle complex interactions and high-dimensional observations. However, these policies are generally limited in horizon length. Our approach, Broadly-Exploring, Local-policy Trees (BELT), merges these two approaches to leverage the strengths of both through a task-conditioned, model-based tree search. BELT uses an RRT-inspired tree search to efficiently explore the state space. Locally, the exploration is guided by a task-conditioned, learned policy capable of performing general short-horizon tasks. This task space can be quite general and abstract; its only requirements are to be sampleable and to well-cover the space of useful tasks. This search is aided by a task-conditioned model that temporally extends dynamics propagation to allow long-horizon search and sequential reasoning over tasks. BELT is demonstrated experimentally to be able to plan long-horizon, sequential trajectories with a goal conditioned policy and generate plans that are robust.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “Machine learning has recently been applied to enable abstract task specification, allow general primitives, or relax constraints [67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78].”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “The combination of learned model-free policies with structures like graphs (Savinov et al., 2018; Eysenbach et al., 2019) and trees (Ichter et al., 2021) to plan over extended horizons has also been demonstrated to improve generalization and exploration.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Alternatively, a number of recent works have studied long horizon visual planning [52, 53, 73, 74, 75, 50, 76, 77, 78], often by combining a structured search approach with learned components like generative models [73, 50, 76] or distance functions [53, 34].”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “Combining planning algorithms with goal-conditioned RL, as in some recent works (Eysenbach et al., 2019; Ichter et al., 2020), could be an exciting direction for addressing this limitation.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “Other prior work has focused on learning representations that can be used for planning [58, 82, 104, 105, 127].”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, methodology): “The resulting image is passed through a convolutional image encoder with ReLU activations with the following architecture [channels, kernel size, stride] (all with padding 1): [[32, 4, 2], [32, 4, 1], [64, 4, 2], [64, 4, 2], [128, 4, 2], [128, 4, 1], [256, 4, 2], [256, 4, 1]].”