RVT: Robotic View Transformer for 3D Object Manipulation
By Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, Dieter Fox. Conference on Robot Learning, 2023
299 citations recorded by the source corpus.
For 3D object manipulation, methods that build an explicit 3D representation perform better than those relying only on camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, a multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism to aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state-of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few ($\sim$10) demonstrations per task. Visual results, code, and trained model are provided at https://robotic-view-transformer.github.io/.
Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/a118acfe34381d4508c2e084f9247c9f7bfc6f48
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “RVT, RVT-2 [105], [106] propose re-rendering images from virtual views of the scene point cloud and using these images as inputs, rather than directly relying on 3D inputs.”
- ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, methodology): “It is compared to a monolithic learning-based baseline based on the transformer architecture [154] adopted from RVT [155, 156].”
- 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “BEV or 3D policy formulations e.g., Transporter Networks [84, 29], C2F-ARM [33], PerAct [69], Act3D [19] and Robot View Transformer [22], discretize the robot’s workspace for localizing the robot’s end-effector, such as the 2D BEV map for pick and place actions [84], the 3D workspace [19], or…”