Instruction-driven history-aware policies for robotic manipulations

By Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, Cordelia Schmid. 2022

153 citations recorded by the source corpus.

In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions. Yet, robotic manipulation is extremely challenging as it requires fine-grained motor control, long-term memory as well as generalization to previously unseen tasks and environments. To address these challenges, we propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations while (iii) keeping track of the full history of observations and actions. Such an approach enables learning dependencies between history and instructions and improves manipulation precision using multiple views. We evaluate our method on the challenging RLBench benchmark and on a real-world robot. Notably, our approach scales to 74 diverse RLBench tasks and outperforms the state of the art. We also address instruction-conditioned tasks and demonstrate excellent generalization to previously unseen variations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Chelsea Finn
3. Pieter Abbeel
4. Li Fei-Fei
5. Yoshua Bengio
6. Quoc V. Le
7. Trevor Darrell
8. Igor Mordatch

In Computer Vision and Pattern Recognition

1. Sergey Levine
2. Luc Van Gool
3. Trevor Darrell
4. Li Fei-Fei
5. Jitendra Malik
6. Dacheng Tao
7. Xiaogang Wang
8. Andrew Zisserman

In Control and Systems Engineering

1. Sergey Levine
2. Pieter Abbeel
3. Chelsea Finn
4. Corey Lynch
5. Pierre Sermanet
6. Karol Hausman
7. Jonathan Tompson
8. Peter R. Florence

In Robotics & Control

1. Sergey Levine
2. Corey Lynch
3. Pierre Sermanet
4. Andy Zeng
5. Tianhe Yu
6. Brian Ichter
7. Peter R. Florence
8. Fei Xia

Citations in the corpus

• PaLM-E: An Embodied Multimodal Language Model (2023, background): “Prior works focus on combining vision and language inputs in an embodied setting with the goal of direct action prediction (Guhur et al., 2022; Shridhar et al., 2022b;a; Zhang & Chai, 2021; Silva et al., 2021; Jang et al., 2022; Nair et al., 2022; Lynch et al., 2022; Brohan et al., 2022).”
• A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Hiveformer [101] places significant emphasis on leveraging multi-view scene observations and maintaining the full observation history for a language-conditioned policy.”
• Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, background): “Hiveformer (Guhur et al., 2023) proposes to jointly model languages, multiple views, and history with a unified transformer model.”
• 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “We also test 3D Diffuser Actor’s ability to learn multi-task policies from real-world demonstrations.”