Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play

By Sainbayar Sukhbaatar, Ilya Kostrikov, Arthur Szlam, Rob Fergus. 2017

374 citations recorded by the source corpus.

We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner. Our scheme pits two versions of the same agent, Alice and Bob, against one another. Alice proposes a task for Bob to complete; and then Bob attempts to complete the task. In this work we will focus on two kinds of environments: (nearly) reversible environments and environments that can be reset. Alice will "propose" the task by doing a sequence of actions and then Bob must undo or repeat them, respectively. Via an appropriate reward structure, Alice and Bob automatically generate a curriculum of exploration, enabling unsupervised training of the agent. When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

A Tight and Unified Analysis of Extragradient for a Whole Spectrum of Differentiable Games (2019, background): “For instance, the actor-critic model (Pfau and Vinyals, 2016), generative adversarial networks (GANs) (Goodfellow et al., 2014) and automatic curricula (Sukhbaatar et al., 2018) can be cast as two-player games.”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “Self-play was shown to be a powerful algorithm for learning skills within high-dimensional continuous environments [33] and a method for automatically generating curricula [34].”
World Models (2018, background): “Recent works about self-play in RL (Sukhbaatar et al., 2017; Bansal et al., 2017; Al-Shedivat et al., 2017) and PowerPlay (Schmidhuber, 2013; Srivastava et al., 2012) also explores methods that lead to a natural curriculum learning (Schmidhuber, 2002), and we feel this is one of the more exciting…”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “• Noise in Parameter Space for Exploration (Fortunato et al., 2018; Plappert et al., 2018): Pertubates network parameters to aid exploration.”
Curiosity-Driven Exploration by Self-Supervised Prediction (2017, background): “Sukhbaatar et al. (2017) generates supervision for pre-training via asymmetric self-play between two agents to improve data efficiency during fine-tuning.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, neutral): “, 2021b), and their own opponents (Sukhbaatar et al., 2017).”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “Note that prior techniques that have formulated adversarial games, such as asymmetric self play (ASP) [42], generally use a single adversary.3 Choice of state representation and the effect of resets.”
Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning (2017, methodology): “Prior methods have studied curriculum generation by maintaining a separate goal setting policy or network (Sukhbaatar et al., 2017; Matiisen et al., 2017; Held et al., 2017).”
Exploration by Random Network Distillation (2018, methodology): “Other methods of exploration include adversarial self-play (Sukhbaatar et al., 2018), maximizing empowerment (Gregor et al., 2017), parameter noise (Plappert et al., 2017; Fortunato et al., 2017), identifying diverse policies (Eysenbach et al., 2018; Achiam et al., 2018), and using ensembles of…”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, background): “The ability to act in mixed cooperative-competitive environments may be critical for intelligent agents; while competitive training provides a natural curriculum for learning [30], agents must also exhibit cooperative behavior (e.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[28] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, methodology): “Unlike prior adversarial unsupervised RL methods (e.g., Sukhbaatar et al. (2017)), DIAYN forms a cooperative game, which avoids many of the instabilities of adversarial saddle-point formulations.”
Hindsight Experience Replay (2017, background): “Another approach investigated by Sukhbaatar et al. (2017) and Held et al. (2017) uses self-play between the policy and a task-setter in order to automatically generate goal states which are on the border of what the current policy can achieve.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Self-play in adversarial groups of agents is common in multi-agent reinforcement learning [44, 46].”
Learning Latent Plans from Play (2019, background): “Goal conditioned policies have been explored extensively in the literature for reinforcement learning [23, 44, 36, 52, 3, 32, 48, 8, 58], as well as for control via inverse models [1, 35, 9, 61].”
Parameter Space Noise for Exploration (2017, background): “In the context of deep reinforcement learning, a large variety of techniques have been proposed to improve exploration [5, 3, 1, 7, 2, 36, 37].”
Emergent Complexity via Multi-Agent Competition (2017, methodology): “Sukhbaatar et al. [2017] introduced a self-play method for generating an automatic training curriculum in single-agent environments.”
Large-Scale Study of Curiosity-Driven Learning (2018, methodology): “Alternative methods of exploration include Sukhbaatar et al. [45] where they utilize an adversarial game between two agents for exploration.”
Reverse Curriculum Generation for Reinforcement Learning (2017, methodology): “Closer to our method of adaptively generating the tasks to train on, an interesting asymmetric selfplay strategy has recently been proposed [18].”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “In Asymmetric self-play (OpenAI et al., 2021; Sukhbaatar et al., 2018), two agents interact in turn in the environment: Alice and Bob.”