Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

By Tony Zhao, Vikash Kumar, Sergey Levine, Chelsea Finn. Robotics: Science and Systems Conference, 2023

2,043 citations recorded by the source corpus.

Fine manipulation tasks, such as threading cable ties or slotting a battery, are notoriously difficult for robots because they require precision, careful coordination of contact forces, and closed-loop visual feedback. Performing these tasks typically requires high-end robots, accurate sensors, or careful calibration, which can be expensive and difficult to set up. Can learning enable low-cost and imprecise hardware to perform these fine manipulation tasks? We present a low-cost system that performs end-to-end imitation learning directly from real demonstrations, collected with a custom teleoperation interface. Imitation learning, however, presents its own challenges, particularly in high-precision domains: errors in the policy can compound over time, and human demonstrations can be non-stationary. To address these challenges, we develop a simple yet novel algorithm, Action Chunking with Transformers (ACT), which learns a generative model over action sequences. ACT allows the robot to learn 6 difficult tasks in the real world, such as opening a translucent condiment cup and slotting a battery with 80-90% success, with only 10 minutes worth of demonstrations. Project website: https://tonyzhaozh.github.io/aloha/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/91eb20f923ea3b0246868902aef4e9bea572b800

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, positive): “Given this temporal structure, decoding latent actions as action chunks [93] is an intuitive choice, aligning the chunk size with the control frequency of the target embodiment.”
AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems (2025, neutral): “By open-sourcing the dataset, tools, and models, we aim to democratize access to large-scale, high-quality robot data, advancing the pursuit of scalable and general-purpose intelligence.”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, methodology): “Indeed, direct behavior cloning can enable robots to learn a variety of primitive robot skills ranging from lane-following in mobile robots [67], to simple pick-and-place manipulation skills [12, 20] to more delicate manipulation skills like spreading pizza sauce or slotting in a battery [18, 104].”
Robotic Control via Embodied Chain-of-Thought Reasoning (2024, positive): “Robotic policies trained end-to-end for outputting raw low-level actions in response to observations provide a powerful and appealing learning-based approach to robotic control, obviating the need for complex sensing and control stacks [1, 2, 3].”
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2024, positive): “H for metric calculation details). methods (Zhao et al., 2023; Brohan et al., 2023; Kim et al., 2024), while simultaneously requiring scalabilty for stable large-scale training on multimodal data (text, vision, actions).”
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators (2023, neutral): “…history and enabled impressive robot capabilities in a diverse range of challenging situations such as fine grain manipulation of everyday tasks [9], robot surgery [10], [11], [12], underwater exploration [13], [14], and disaster relief [15], [16], all of which would be useful skills for a…”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, positive): “The success of transformer models [58] in natural language processing and computer vision motivated a number of recent works that collected large-scale demonstration datasets and trained transformer-based policies on them [2, 16, 25, 38, 39, 42, 49, 51, 65, 67].”
BridgeData V2: A Dataset for Robot Learning at Scale (2023, methodology): “Prior work on robot learning has often leveraged small datasets that cover a single task in a single domain to develop better learning methods [10, 11, 12, 13, 14, 15, 16, 17, 18].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “ACT [119] builds a conditional VAE policy with action chunking, requiring the policy to predict a sequence of actions rather than a single one.”
HumanPlus: Humanoid Shadowing and Imitation from Humans (2024, methodology): “…our imitation learning method Humanoid Imitation Transformer with three baseline methods: HIT policies with monocular inputs ( Monocular ), ACT [104], and Open-loop trajectory replay, across all tasks: Fold Clothes , Rear-range Objects , Type “AI" , Two-Robot Greeting , Ware-house , and Wear a…”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, positive): “Additionally, inspired by recent advances in robot learning [8, 14, 71], we predict sequences of actions (action chunking) rather than a single action at each timestep.”
Octo: An Open-Source Generalist Robot Policy (2024, methodology): “• Using action chunking : We found it helpful to use “action chunking” [98], i.e., to predict multiple actions into the future, for getting more coherent policy movements.”
RVT: Robotic View Transformer for 3D Object Manipulation (2023, background): “The success of transformers in vision and NLP has led its way into robot learning [42, 43, 44, 17, 45, 46].”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, neutral): “One straightforward strategy is to execute these actions consecutively (Action Chunking [75]) based on the current observation, o t .”
Open-TeleVision: Teleoperation with Immersive Active Visual Feedback (2024, neutral): “Various teleoperation approaches have been studied using VR devices [3, 2], RGB cameras [4, 5, 6], wearable gloves [7, 8, 9], and customized hardwares [10, 11].”
π0: A Vision-Language-Action Flow Model for General Robot Control (2024, neutral): “We tried temporal ensembling [57] early on and found that it hurt policy performance, so we opted not to aggregate actions and instead execute action chunks open-loop.”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, neutral): “For parallel decoding, we replace this input with empty action embeddings that differ only in their positional encoding values (similar to [53]).”
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (2024, methodology): “In this paper, we use Diffusion Policy [9] for all of our experiments, while other frameworks such as ACT [53] could potentially serve as a drop-in replacement.”