Solving Rubik's Cube with a Robot Hand

By OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Ma-teusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell and 9 more. 2019

1,497 citations recorded by the source corpus.

We demonstrate that models trained only in simulation can be used to solve a manipulation problem of unprecedented complexity on a real robot. This is made possible by two key components: a novel algorithm, which we call automatic domain randomization (ADR) and a robot platform built for machine learning. ADR automatically generates a distribution over randomized environments of ever-increasing difficulty. Control policies and vision state estimators trained with ADR exhibit vastly improved sim2real transfer. For control policies, memory-augmented models trained on an ADR-generated distribution of environments show clear signs of emergent meta-learning at test time. The combination of ADR with our custom robot platform allows us to solve a Rubik's cube with a humanoid robot hand, which involves both control and state estimation problems. Videos summarizing our results are available: this https URL

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Scaling Laws for Transfer (2021, background): “Another notable work that helped motivate our investigation into transfer was sim-to-real transfer training for solving a Rubik’s cube with a robot hand [OAA + 19], a setting in which the ﬁne-tuning data is far more expensive than the pre-training data.”
Shortcut learning in deep neural networks (2020, methodology): “s a consequence of narrow shortcut learning by adapting to speciﬁc details of the simulation. Introducing additional variation in colour, size, texture, lighting, etc. helps a lot in closing this gap [70,71]. Fairness & algorithmic decision-making Tasked to predict strong candidates on the basis”
Augmented Language Models: a Survey (2023, methodology): “…with an environment, and it has been used for applications such as playing games (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019; Team et al., 2021; Bakhtin et al., 2022) or controlling robots (Gu et al., 2017; Kalashnikov et al., 2018; Akkaya et al., 2019; Lee et al., 2020).”
The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, methodology): “OpenAI's Rubik's solver (OpenAI et al., 2019) is (although it was not pitched as such) is a hybrid of a symbolic algorithm for solving the cognitive aspects of a Rubik's cube, and deep reinforcement learning for the manual manipulation aspects.”
Towards Causal Representation Learning (2021, background): “accumulating evidence supporting the usefulness of an appropriate structured representation of the environment [2, 26, 258].”
IIRC: Incremental Implicitly-Refined Classification (2020, background): “Deep learning algorithms have led to transformational breakthroughs in computer vision [12, 17], natural language processing [19, 50], speech processing [3, 5], reinforcement learning [36, 44], robotics [16, 1], recommender”
The MAGICAL Benchmark for Robust Imitation (2020, neutral): “This can be viewed as a post-hoc form of domain randomisation, which has previously yielded impressive results in robotics and RL [2].”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, methodology): “For continuous control tasks, on-policy algorithms such as REINFORCE [2], TRPO [10], A3C [14], PPO [17] and off-policy algorithms such as DDPG [13] and SAC [21] have enabled successful applications such as quadrupedal locomotion [20], self-driving [30] or dexterous in-hand manipulation [20, 25, 32].”
Reinforcement Learning with Augmented Data (2020, background): “Real-world successes of reinforcement learning often require months of data-collection and (or) training [6, 9].”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Recently, a number of end-to-end frameworks have been proposed for vision-based manipulation [2, 3, 4, 5].”
Multi-Task Learning with Deep Neural Networks: A Survey (2020, background): “In recent years, many of the advances in reinforcement learning have focused on optimization and training methods (Schulman et al., 2017; Haarnoja et al., 2018; Akkaya et al., 2019).”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, methodology): “Previously, the PPO algorithm has been effectively applied across a diverse range of domains, such as computer vision [50, 51, 52] and robotics [53, 54, 55].”