Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

By Mohit Shridhar, Lucas Manuelli, Dieter Fox. 2022

837 citations recorded by the source corpus.

Transformers have revolutionized vision and natural language processing with their ability to scale with large datasets. But in robotic manipulation, data is both limited and expensive. Can manipulation still benefit from Transformers with the right problem formulation? We investigate this question with PerAct, a language-conditioned behavior-cloning agent for multi-task 6-DoF manipulation. PerAct encodes language goals and RGB-D voxel observations with a Perceiver Transformer, and outputs discretized actions by ``detecting the next best voxel action''. Unlike frameworks that operate on 2D images, the voxelized 3D observation and action space provides a strong structural prior for efficiently learning 6-DoF actions. With this formulation, we train a single multi-task Transformer for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task. Our results show that PerAct significantly outperforms unstructured image-to-action agents and 3D ConvNet baselines for a wide range of tabletop tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Model Architectures, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Model Architectures

In Robotics & Control

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, background): “Prior works focus on combining vision and language inputs in an embodied setting with the goal of direct action prediction (Guhur et al., 2022; Shridhar et al., 2022b;a; Zhang & Chai, 2021; Silva et al., 2021; Jang et al., 2022; Nair et al., 2022; Lynch et al., 2022; Brohan et al., 2022).”
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills (2023, background): “For example, Shridhar et al. (2022) modiﬁed 18 tasks in James et al. (2020) to enable few variations in initial states.”
ChatGPT for Robotics: Design Principles and Model Abilities (2023, background): “These systems are often coupled with additional features from pretrained large-scale vision and language models models [11], [14], [15], [16], [37], [38], [39].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Other works have incorporated pre-trained language models, often either as an instruction encoder [63, 64, 43, 41, 44, 1, 65] or for high-level planning [66, 19, 17, 67, 68, 69].”
Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “To address this limitation, the authors of PerAct (Perceiver-Actor) [27] propose to represent observation and action spaces with 3D voxels and employ the 3D structure of voxel patches for efﬁcient language-conditioned behavioral cloning with transformers to imitate 6-DoF manipulation tasks from…”
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (2023, methodology): “CLIPort (Shafiullah et al., 2022) and PerAct (Shridhar et al., 2023) learn language-conditioned manipulation by leveraging CLIP (Rad-ford et al., 2021) to connect visual signals and language instructions.”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, positive): “The success of transformer models [58] in natural language processing and computer vision motivated a number of recent works that collected large-scale demonstration datasets and trained transformer-based policies on them [2, 16, 25, 38, 39, 42, 49, 51, 65, 67].”
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023, background): “Other works emphasize the multi-task or few-shot aspect of imitation learning [14, 25, 11], leveraging language [51, 52, 26, 7], or exploiting the specific task structure [43, 68, 28, 52].”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, background): “…also focus on multi-task or few-shot imitation learning, [25, 27, 30, 34, 46, 50, 88, 102], language-conditioned imitation learning [12, 47, 82, 83], imitation from play data [21, 57, 74, 89], using human videos [16, 24, 29, 60, 69, 80, 84, 96], and using task-specific structures [49, 83, 103].”
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (2024, neutral): “If we distinguish the existing work with sources of demonstrations, we can classify them into learning from real-robot expert data [14, 10, 12, 59, 60, 61, 62, 50, 63, 1, 64, 51, 65], learning from play data [66, 67, 68], and learning from human demonstrations [67, 69, 70, 71, 72, 73, 74, 56].”
RVT: Robotic View Transformer for 3D Object Manipulation (2023, methodology): “Similar to PerAct, we apply For Image-BC and C2F-ARM-BC, we adopt the evaluation results from [6] since their trained models have not been released.”
Text2Motion: from natural language instructions to feasible plans (2023, background): “While many methods learn short-horizon skills [52, 28, 45, 46, 47, 29], others focus on long-horizon tasks [40, 7].”
Robotic Control via Embodied Chain-of-Thought Reasoning (2024, background): “While some works use strong structural priors in their policies to enable this integration [43, 44, 45], vision-language-action models (VLAs) have recently been proposed as a simple yet scalable alternative [5, 6, 7], achieving state-of-the-art performance for generalist robot policies [7] and…”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, neutral): “, 2021), Perceiver-Actor (Shridhar et al., 2022), and RT-1 (Brohan et al.”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, background): “Although recent years have seen several large multi-task robot policies proposed in the literature [51, 23], such models often have limited breadth of real-world tasks, as with Gato [51], or focus on training tasks rather than generalization to new tasks, as with recent instruction following methods”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, neutral): “In recent years, there has been a surge of interest in robotic control models equipped with visual capabilities [7, 8, 15, 30, 34, 45, 48, 58, 60, 62, 67, 69].”
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play (2023, methodology): “Noticeably, there are other approaches used for improving sample efficiency for imitation learning, such as grounding action on discrete observation space [23, 44, 50, 51].”
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (2023, neutral): “Commonly used teleoperation systems include joysticks and spacemouses [20], [21], [22], commercial VR controllers [4], [23], [24], [25], RGB cameras [26], [27], [28], [29] or IMU sensors [30], [31], [32], However due to the morphological differences between these control devices and the robots, the…”