Run the GAMUT: a comprehensive approach to evaluating game-theoretic algorithms

By Eugene Nudelman, Jennifer Wortman Vaughan, Yoav Shoham, Kevin Leyton-Brown. 2004

224 citations recorded by the source corpus.

We present GAMUT^1, a suite of game generators designed for testing game-theoretic algorithms. We explain why such a generator is necessary, offer a way of visualizing relationships between the sets of games supported by GAMUT, and give an overview of GAMUTýs architecture. We highlight the importance of using comprehensive test data by benchmarking existing algorithms. We show surprisingly large variation in algorithm performance across different sets of games for two widely-studied problems: computing Nash equilibria and multiagent learning in repeated games.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Empirically Evaluating Multiagent Learning Algorithms (2014, methodology): “We selected thirteen game generators from the GAMUT game collection; these are summarized in Table X. Details of each generator are available in GAMUT’s online documentation; see gamut.stanford.edu .”
New Criteria and a New Algorithm for Learning in Multi-Agent Systems (2004, methodology): “Let us first consider the results of a tournament over a full set of games in GAMUT.”
A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity (2017, background): “Also, it is common to evaluate learning algorithms in randomly generated games according to certain specifications such as zero-sum games or, constant-sum games (Nudelman et al., 2004).”
An Overview of Cooperative and Competitive Multiagent Learning (2005, background): “The AI agenda however allows for a lively competition possibility by introducing open competition on the extensive list of games generated, for example, in the GAMUT framework [117].”
Simple search methods for finding a Nash equilibrium (2004, methodology): “To this end we use GAMUT, a recently introduced computational testbed for game theory (Nudelman et al. 2004).”
Computing the optimal strategy to commit to (2006, methodology): “First, we can empirically evaluate the techniques presented here on test suites such as GAMUT[19].”
Learning against multiple opponents (2006, background): “testing environment first described in [21] by testing against a number of existing approaches from the multi-agent learning literature over a wide variety of repeated games from GAMUT [18].”
Evolutionary Tournament-Based Comparison of Learning and Non-Learning Algorithms for Iterated Games (2007, background): “There are other testbeds for arbitrary number of players, arbitrary number of actions (Nudelman et al 2004), but the class of conflicting two-action, two-player games provides a sufficiently rich environment for our purpose.”