A Study of AI Population Dynamics with Million-agent Reinforcement Learning

By Yaodong Yang, Lantao Yu, Yiwei Bai, Ying Wen, Weinan Zhang, Jun Wang. 2017

41 citations recorded by the source corpus.

We conduct an empirical study on discovering the ordered collective dynamics obtained by a population of intelligence agents, driven by million-agent reinforcement learning. Our intention is to put intelligent agents into a simulated natural context and verify if the principles developed in the real world could also be used in understanding an artificially-created intelligent population. To achieve this, we simulate a large-scale predator-prey world, where the laws of the world are designed by only the findings or logical equivalence that have been discovered in nature. We endow the agents with the intelligence based on deep reinforcement learning (DRL). In order to scale the population size up to millions agents, a large-scale DRL training platform with redesigned experience buffer is proposed. Our results show that the population dynamics of AI agents, driven only by each agent's individual self-interest, reveals an ordered pattern that is similar to the Lotka-Volterra model studied in population biology. We further discover the emergent behaviors of collective adaptations in studying how the agents' grouping behaviors will change with the environmental resources. Both of the two findings could be explained by the self-organization theory in nature.

Subfields: Economics and Econometrics, Sociology and Political Science, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

1. Thore Graepel
2. Joel Z. Leibo
3. Julien Pérolat

In Sociology and Political Science

1. Thore Graepel
2. Joel Z. Leibo
3. Julien Pérolat
4. Edward Hughes

In Statistical and Nonlinear Physics

1. Y. Liu
2. Thore Graepel
3. Joel Z. Leibo
4. Adam Henry Marblestone
5. Edward Hughes

Citations in the corpus

• From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, background): “Furthermore, in large-scale multiagent systems, the dynamics and stochasticity of the environment become more complex, which makes it more challenging to achieve coordination among agents (Singh, Jain, and Sukhbaatar 2019; Yang et al. 2018a; Jiang and Lu 2018).”
• Malthusian Reinforcement Learning (2018, background): “Similar oscillations in subpopulation sizes were recently observed in a large-scale multi-agent learning simulation by [41].”
• Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents (2019, unknown): “Most such works further focus on learning a specific dynamic, such as predator-prey Yang et al. (2018b) or are more concerned with the study than the learning of behavior, and use hard-coded rewards Zheng et al. (2017).”