Transporter Networks: Rearranging the Visual World for Robotic Manipulation

By Andy Zeng, Peter R. Florence, Jonathan Tompson, Stefan Welker, Jonathan M. Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, D. Duong, Dan Duong and 2 more. 2020

585 citations recorded by the source corpus.

Robotic manipulation can be formulated as inducing a sequence of spatial displacements: where the space being moved can encompass an object, part of an object, or end effector. In this work, we propose the Transporter Network, a simple model architecture that rearranges deep features to infer spatial displacements from visual input - which can parameterize robot actions. It makes no assumptions of objectness (e.g. canonical poses, models, or keypoints), it exploits spatial symmetries, and is orders of magnitude more sample efficient than our benchmarked alternatives in learning vision-based manipulation tasks: from stacking a pyramid of blocks, to assembling kits with unseen objects; from manipulating deformable ropes, to pushing piles of small objects with closed-loop feedback. Our method can represent complex multi-modal policy distributions and generalizes to multi-step sequential tasks, as well as 6DoF pick-and-place. Experiments on 10 simulated tasks show that it learns faster and generalizes better than a variety of end-to-end baselines, including policies that use ground-truth object poses. We validate our methods with hardware in the real world. Experiment videos and code will be made available at this https URL

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “This instantiation of Inner Monologue uses (i) InstructGPT [9, 91] as the LLM for multi-step planning [20, 21], (ii) scripted modules to provide language feedback in the form of object recognition ( Object ), success detection ( Success ), and task-orogress scene description ( Scene ), and (iii) a p”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “, models similar to CLIPort (Shridhar et al., 2022; Zeng et al., 2020) for open vocabulary pick-and-place).”
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “There are several approaches to ofﬂine imitation learning [31–34, 9, 10, 25, 35, 36] and ofﬂine reinforcement learning [26, 27, 37–43] (see Appendix A for more discussion on related work).”
Implicit Behavioral Cloning (2021, methodology): “In real-world robotics research, BC has been widely used in policy learning [4, 30, 5, 25].”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, methodology): “To study these benefits, we conduct large-scale experiments in the Ravens [2] framework with a simulated suction-gripper robot.”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “…et al., 2020), due to the difﬁculty in parameterizing different scenes and tasks through a single network (Wang et al., 2018b; Jiang et al., 2019; Zeng et al., 2020) as well as the lack of large off-the-shelf datasets for pre-training (Cobbe et al., 2020; Zhu et al., 2020; Yu et al., 2020).”
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, methodology): “CLIPort [42] uses a frozen CLIP vision and text encoders in combination with Transporter networks [47] for imitation learning.”
Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, methodology): “9M parameter FCN [41] with residual connections [25], given that similar architectures have been used for FCN-based picking and placing [71], [68].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “CLIPort [43] integrates the vision and language encoders of CLIP [31] with the Transporter network [94], creating a two-stream architecture.”
Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “A popular approach to model multimodal action distributions while maintaining the simplicity of direction action mapping is convert the regression task into classi ﬁ cation by discretizing the action space (Zeng et al., 2021; Wu et al., 2020; Avigal et al., 2022).”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, background): “…also focus on multi-task or few-shot imitation learning, [25, 27, 30, 34, 46, 50, 88, 102], language-conditioned imitation learning [12, 47, 82, 83], imitation from play data [21, 57, 74, 89], using human videos [16, 24, 29, 60, 69, 80, 84, 96], and using task-specific structures [49, 83, 103].”
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023, background): “Other works emphasize the multi-task or few-shot aspect of imitation learning [14, 25, 11], leveraging language [51, 52, 26, 7], or exploiting the specific task structure [43, 68, 28, 52].”
Composing Ensembles of Pre-trained Models via Iterative Consensus (2022, methodology): “In robot manipulation, we use the proposed method to manipulate objects in Ravens (Zeng et al., 2020) to conform to a set of object relations specified by text descriptions or real-world images.”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “For example, CLIPORT [165] and Transporter Networks [223] utilized Pybullet simulator data for training and successfully transferred models to real-world applications.”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, methodology): “We also attempted to benchmark with recently proposed TransporterNets [48].”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, background): “They mainly focus on robotics [15, 44, 48, 40], some with a particular focus on human data [31, 32, 39].”
Large Batch Simulation for Deep Reinforcement Learning (2021, background): “…game engines (Kolve et al., 2017; Dosovitskiy et al., 2017; Lee et al., 2019; Gan et al., 2020; James et al., 2020), and those built from scratch using open-source 3D graphics and physics libraries (Savva et al., 2017; 2019; Xia et al., 2018; 2020; Xiang et al., 2020; Zeng et al., 2020).”
RVT: Robotic View Transformer for 3D Object Manipulation (2023, methodology): “These view-based methods have achieved impressive success on a variety of pick-and-place and object rearrangement tasks [1, 2, 3, 4].”
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023, methodology): “However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3–7,14,18,19,22,30,33,51, 55, 56, 64–66], making the system broadly applicable.”