Google Research Football: A Novel Reinforcement Learning Environment

By Karol Kurach, Anton Raichuk, Piotr Sta'nczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet and 1 more. 2019

523 citations recorded by the source corpus.

Recent progress in the field of reinforcement learning has been accelerated by virtual learning environments such as video games, where novel algorithms and ideas can be quickly tested in a safe and reproducible manner. We introduce the Google Research Football Environment, a new reinforcement learning environment where agents are trained to play football in an advanced, physics-based 3D simulator. The resulting environment is challenging, easy to use and customize, and it is available under a permissive open-source license. In addition, it provides support for multiplayer and multi-agent experiments. We propose three full-game scenarios of varying difficulty with the Football Benchmarks and report baseline results for three commonly used reinforcement algorithms (IMPALA, PPO, and Ape-X DQN). We also provide a diverse set of simpler scenarios with the Football Academy and showcase several promising research directions.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “Testbed Environments: We evaluate the performance of MAPPO and IPPO on four cooperative benchmark – the multi-agent particle-world environment (MPE), the StarCraft multi-agent challenge (SMAC), the Hanabi challenge, and Google Research Football (GRF) – and compare these methods’ performance to popul”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “Even much work has been done recently (Kurach et al., 2019; Li et al., 2021; Liu et al., 2021b), there remain many problems in building agents for the GRF: (1) Multiple Players: In the GRF, there are both cooperative and competitive players.”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “Moreover, for GRF scenarios, we use a prioritized replay buffer of the TD error for CDS, all the baselines, and ablations.”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “[17] that require massively parallelized training, STS2 is a lightweight benchmark where MARL agents can outperform the scripted opponent team within hours on a single CPU.”
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, methodology): “On top of them, we also test MAT on the Bimanual Dexterous Hands Manipulation (Bi-DexHands) [6] which provides a list of challenging bimanual manipulation tasks (see Figure (3)), and the Google Research Football [18] benchmark with a series of cooperation scenarios in football game.”
A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “Google research football [100] is an even more challenging game with a physics-based 3D soccer simulator.”
Promoting Coordination through Policy Regularization in Multi-Agent Deep Reinforcement Learning (2019, methodology): “Finally, we verify that these beneﬁts hold on the more complex, discrete action, Google Research Football environment [20].”
Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning (2019, background): “We refer the readers to (Kurach et al. 2019) for the details.”
Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games (2021, methodology): “In addition to the experiments on relatively simple games, we also evaluate our methods on a challenging real-world game named Google Research Football (GRF) [14].”