Skill Rating for Generative Models

By Catherine Olsson, Surya Bhupatiraju, Tom B. Brown, Augustus Odena, Ian J. Goodfellow. 2018

35 citations recorded by the source corpus.

We explore a new way to evaluate generative models using insights from evaluation of competitive games between human players. We show experimentally that tournaments between generators and discriminators provide an effective way to evaluate generative models. We introduce two methods for summarizing tournament outcomes: tournament win rate and skill rating. Evaluations are useful in different contexts, including monitoring the progress of a single model as it learns during the training process, and comparing the capabilities of two different fully trained models. We show that a tournament consisting of a single model playing against past and future versions of itself produces a useful measure of training progress. A tournament containing multiple separate models (using different seeds, hyperparameters, and architectures) provides a useful relative comparison between different trained GANs. Tournament-based rating methods are conceptually distinct from numerous previous categories of approaches to evaluation of generative models, and have complementary advantages and disadvantages.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Economics and Econometrics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Economics and Econometrics

In Evaluation & Benchmarks

Citations in the corpus

Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, methodology): “Following previous work (Olsson et al. 2018), in our paper, we use the Glicko2 system (Glickman 2012).”
Self-Attention Generative Adversarial Networks (2018, background): “Though alternatives exist (Zhou et al., 2019; Khrulkov & Oseledets, 2018; Olsson et al., 2018), they are not widely used.”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, methodology): “Evaluating a model by estimating its optimal classification error has been considered by several earlier works (Olsson et al., 2018; Kannan and Vinyals, 2016; Li et al., 2017; Bruni and Fernandez, 2017; Bowman et al., 2016).”
Discriminator Rejection Sampling (2018, background): “Moreover, trained GAN discriminators aren’t just identically 1/2 — they can even be used to perform chess-type skill ratings of other trained generators (Olsson et al., 2018).”
Fastpitch: Parallel Text-to-Speech with Pitch Prediction (2020, background): “It is known in the context of automatic scoring of generative models [17].”
Pros and Cons of GAN Evaluation Measures (2018, methodology): “Skill Rating: Here the idea is to use a skill rating system to summarize tournament outcomes in a way that takes into account the amount of new information each match provides.”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, background): “The closest recent work to ours compares models using a tournament of discriminators [42].”
Improved Consistency Regularization for GANs (2020, background): “…be hard to train for a variety of reasons, there is a substantial amount of work (Metz et al., 2016; Arjovsky et al., 2017; Gulrajani et al., 2017; Olsson et al., 2018; Sinha et al., 2019) dedicated to ﬁxing these issues, understanding them better, or more accurately measuring the quality of GAN…”
A Domain Agnostic Measure for Monitoring and Evaluating GANs (2018, methodology): “In [26], two metrics are introduced to evaluate a single model playing against past and future versions of itself, as well as to measure the aptitude of two different fully trained models.”
Evaluating GANs via Duality (2018, unknown): “…explored approximations of DG, where instead of using optimization to ﬁnd the optimal v worst and u worst , we limit the search space to a set of discriminators and generators that are stored as snapshots throughout the training, similarly to (Olsson et al., 2018) (see results in Appendix B.3).”