Emergence of Locomotion Behaviours in Rich Environments

By Nicolas Manfred Otto Heess, TB Dhruva, S. Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyun Wang, S. M. Ali Eslami and 2 more. 2017

1,015 citations recorded by the source corpus.

The reinforcement learning paradigm allows, in principle, for complex behaviours to be learned directly from simple reward signals. In practice, however, it is common to carefully hand-design the reward function to encourage a particular solution, or to derive it from demonstration data. In this paper explore how a rich environment can help to promote the learning of complex behavior. Specifically, we train agents in diverse environmental contexts, and find that this encourages the emergence of robust behaviours that perform well across a suite of tasks. We demonstrate this principle for locomotion -- behaviours that are known for their sensitivity to the choice of reward. We train several simulated bodies on a diverse set of challenging terrains and obstacles, using a simple reward function based on forward progress. Using a novel scalable variant of policy gradient reinforcement learning, our agents learn to run, jump, crouch and turn as required by the environment without explicit reward-based guidance. A visual depiction of highlights of the learned behavior can be viewed following this https URL .

Subfields: Artificial Intelligence, Biomedical Engineering, Cell Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Cell Biology

In Reinforcement Learning

Citations in the corpus

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, background): “A popular class of on-policy algorithms is policy gradient methods (Williams, 1992; Sutton et al., 2000), which have been shown to be effective for a diverse array of complex tasks (Heess et al., 2017; Pathak et al., 2017; Peng et al., 2018; Rajeswaran et al., 2018).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “These techniques have recently become popular in RL (Wang et al., 2016; Heess et al., 2017; Schulman et al., 2017b), although in terms of final reward performance on continuous control benchmarks, TRPO is still considered to be the state-of-the-art.”
Distributed Prioritized Experience Replay (2018, background): “, 2017), Distributed PPO (Heess et al., 2017) and AlphaGo (Silver et al.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “In terms of manipulation, the tasks being considered for evaluating DRL algorithms are more standardized in the recent literature (Lillicrap et al., 2015; Schulman et al., 2015a; Mnih et al., 2016; Heess et al., 2017; Wu et al., 2017).”
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, background): “Heess et al. [29] showed that complex locomotion behaviors, such as running, jumping and crouching, can be learned directly from simple reward signals in rich simulated environments.”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, background): “Such unnatural behaviors are indeed quite prevalent in the recent DRL results [15].”
Critic Regularized Regression (2020, background): “[13] Nicolas Heess, Dhruva Tirumala, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “, 2019), and learning motor control for high-dimensional simulated robots (Akkaya et al., 2019; Heess et al., 2017).”
Benchmarking Model-Based Reinforcement Learning (2019, methodology): “While model-free algorithms have achieved success in areas including robotics [30, 44, 20, 1], video-games [34, 33], and motion animation [38], their high sample complexity limits largely their application to simulated domains.”
ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “To leverage demonstrations during reinforcement learning, we initialize training episodes from states sampled randomly from the demonstration trajectories and do policy learning using a distributed implementation of proximal policy optimization (PPO) [43, 44].”
Maximum a Posteriori Policy Optimisation (2018, methodology): “Recent experiments (Mnih et al., 2015; Jader-berg et al., 2016; Heess et al., 2017) successfully combined these algorithms with powerful deep neural-network approximators while beneﬁting from the increase of compute capacity.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[4] Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al.”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, background): “, 2016), and complex continuous control tasks involving locomotion (Lillicrap et al., 2016; Schulman et al., 2015; 2017; Heess et al., 2017).”
Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, background): “TRPO [42] and PPO [43] hold appeal due to their robustness to hyperparameter settings as well as their scalability [14] but the lack of sample efficiency makes”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, background): “The framework has enabled simulated agents to develop highly dynamic motor skills [4], [5], [6], [7].”
Distributed Distributional Deterministic Policy Gradients (2018, methodology): “Finally, we consider the parkour domain introduced by (Heess et al., 2017).”
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (2017, neutral): “Instead of the hard constraint imposed by the trust region optimizer, Heess et al. [9] and Schulman et al. [24] added a KL cost to the objective function as a soft constraint.”
Emergent Complexity via Multi-Agent Competition (2017, methodology): “Our distributed PPO implementation is similar to the implementation of Heess et al. [2017], where instead of the KL penalty we used the clipped objective as proposed in PPO [Schulman et al., 2017]. Figure 2 shows a high-level view of the architecture. Essentially, we do multiple rollouts in parallel”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “Deep reinforcement learning (RL) enables autonomous learning of diverse and complex tasks with rich sensory inputs, temporally extended goals, and challenging dynamics, such as discrete gameplaying domains [46, 62], and continuous control domains including locomotion [59, 31] and manipulation [56, 3”