Robots that can adapt like animals

By Antoine Cully, Jeff Clune, Danesh S. Tarapore, Jean-Baptiste Mouret. 2014

1,150 citations recorded by the source corpus.

Robots have transformed many industries, most notably manufacturing, and have the power to deliver tremendous benefits to society, such as in search and rescue, disaster response, health care and transportation. They are also invaluable tools for scientific exploration in environments inaccessible to humans, from distant planets to deep oceans. A major obstacle to their widespread adoption in more complex environments outside factories is their fragility. Whereas animals can quickly adapt to injuries, current robots cannot ‘think outside the box’ to find a compensatory behaviour when they are damaged: they are limited to their pre-specified self-sensing abilities, can diagnose only anticipated failure modes, and require a pre-programmed contingency plan for every type of potential damage, an impracticality for complex robots. A promising approach to reducing robot fragility involves having robots learn appropriate behaviours in response to damage, but current techniques are slow even with small, constrained search spaces. Here we introduce an intelligent trial-and-error algorithm that allows robots to adapt to damage in less than two minutes in large search spaces without requiring self-diagnosis or pre-specified contingency plans. Before the robot is deployed, it uses a novel technique to create a detailed map of the space of high-performing behaviours. This map represents the robot’s prior knowledge about what behaviours it can perform and their value. When the robot is damaged, it uses this prior knowledge to guide a trial-and-error learning algorithm that conducts intelligent experiments to rapidly discover a behaviour that compensates for the damage. Experiments reveal successful adaptations for a legged robot injured in five different ways, including damaged, broken, and missing legs, and for a robotic arm with joints broken in 14 different ways. This new algorithm will enable more robust, effective, autonomous robots, and may shed light on the principles that animals use to adapt to injury.

Subfields: Artificial Intelligence, Biomedical Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “[8] demonstrates that exploration in the real world can be more efficient with prior knowledge from a simulator.”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, methodology): “Then in each iteration, MAP-Elites, like general EA algorithm, chose a random organism, mutated them randomly, and replaced with the current ones if the new ones have higher fitness (high certainty for a class of a neural network).”
Solving Rubik's Cube with a Robot Hand (2019, neutral): “This is in contrast to prior work that explicitly searched over a set of policies until it found one that works for a broken robot [22].”
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, methodology): “Popular optimization methods for locomotion control include black-box [12] and Bayesian optimization [13].”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “RPG is also inspired by the map-elite algorithm (Cully et al., 2015) from evolutionary learning community, which optimizes multiple objectives simultaneously for sufﬁciently diverse polices.”
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments (2017, background): “The goal of the agent is to learn to adapt from episode to episode by changing its gait so that it is able to move with a maximal speed in a given direction despite the changes in the environment [cf. 48, 49]..”
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning (2017, methodology): “being high performing), seeking to collect a set of highperforming, yet interestingly different policies (Mouret & Clune, 2015; Lehman & Stanley, 2011b; Cully et al., 2015; Pugh et al., 2016).”
The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities (2018, neutral): “Yet both philosophical arguments [23, 26, 27, 82, 109] and high-profile publications [2, 16, 18 – 21, 52, 59, 60, 83, 85, 86, 92, 141, 146, 148] suggest that digital evolution can be a useful tool to aid and complement the study of biological evolution.”
Curiosity Driven Exploration of Learned Disentangled Goal Spaces (2018, background): “This is key for acquiring world models and repertoires of parameterized skills [3, 4, 5], to efficiently bootstrap exploration for deep reinforcement learning problems with rare or deceptive rewards [6, 7], or to quickly repair strategies in case of damages [8].”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, background): “effectively co-evolve, novelty search with local competition [33], MAP-Elites [34, 35], Innovation Engines [36], which exploit the opportunity to transfer high-quality solutions from one objective among many to another, and the combinatorial multi-objective evolutionary algorithm (CMOEA), which exte”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, methodology): “The idea of preserving and exploring from stepping stones in an archive comes from the quality diversity (QD) family of algorithms (like MAP-elites [60, 79] and novelty search with local competition [80]), and Go-Explore is an enhanced QD algorithm based on MAP-Elites.”
Meta Reinforcement Learning with Latent Variable Gaussian Processes (2018, background): “Although meta learning given multiple or non-stationary dynamics has also been considered in [11, 18, 4, 1].”
Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents (2017, methodology): “That is the approach of novelty search (Lehman & Stanley, 2011a) and quality diversity algorithms (Cully et al., 2015; Mouret & Clune, 2015; Pugh et al., 2016), which are described in detail below.”
Increasing generality in machine learning through procedural content generation (2019, background): “The Map-Elites algorithm in particular, originally introduced to create more robust robot gaits [59], has been adapted to create sets of game levels that vary in what skills they require from the agent or what mechanics they feature [60].”
Expected Policy Gradients (2017, background): “An off-the-shelf kinematics simulator [5] models an arm with three joints, each of which can be set by the agent, whose goal is to get the end effector to a specific location.”
Data-efficient neuroevolution with kernel-based surrogate models (2018, methodology): “teraction that is possible with the physical system. A common approach to optimization in computationally expensive domains is to use approximate models of the objective function, or surrogate models [2, 8, 10, 12]. These models are created through arXiv:1804.05364v2 [cs.NE] 17 Apr 2018 GECCO ’18, J”
Online Learning with Stochastic Recurrent Neural Networks using Intrinsic Motivation Signals (2017, background): “The approach can be used for exploration of unknown environments, learning of unknown workspace and body constraints, or for updating robot models, for example, if a joint gets broken [21].”
Black-box data-efficient policy search for robotics (2017, background): “It can drastically decrease the evaluation time when optimizing gaits [10] or when finding compensatory behaviors for damaged robots [2].”