PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks

By Jiankai Sun, De-An Huang, Bo Lu, Yunhui Liu, Bolei Zhou, Animesh Garg. 2021

68 citations recorded by the source corpus.

In this work, we study the problem of how to leverage instructional videos to facilitate the understanding of human decision-making processes, focusing on training a model with the ability to plan a goal-directed procedure from real-world videos. Learning structured and plannable state and action spaces directly from unstructured videos is the key technical challenge of our task. There are two problems: first, the appearance gap between the training and validation datasets could be large for unstructured videos; second, these gaps lead to decision errors that compound over the steps. We address these limitations with Planning Transformer (PlaTe), which has the advantage of circumventing the compounding prediction errors that occur with single-step models during long model-based rollouts. Our method simultaneously learns the latent state and action information of assigned tasks and the representations of the decision-making process from human demonstrations. Experiments conducted on real-world instructional videos show that our method can achieve a better performance in reaching the indicated goal than previous algorithms. We also validated the possibility of applying procedural tasks on a UR-5 platform. Please see 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “, CLIP [71]) have been trained on joint image(s) and corresponding text captions via variants of a masked language modeling objective [72, 73, 74, 75], a contrastive loss [76, 77, 71] or other supervised objectives[78, 79].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “Simulators with both a high visual fidelity and realistic physics are highly sought after by the robotics communities as they provide the ideal test-bed for various robotic tasks such as navigation and interaction tasks [110], [111].”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “Prior to the emergence of foundation models, traditional deep learning models for robotics were typically trained on limited datasets gathered for distinct tasks [7].”