TrueSkill™: A Bayesian Skill Rating System

By Bernhard Schölkopf, John C. Platt, Thomas Hofmann. 2007

536 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Economics and Econometrics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Evaluation & Benchmarks

Citations in the corpus

Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, methodology): “…quality of NLG models based on pairwise comparison in sample-level, we adopt the skill rating system similar to ELO (Elo 1978) and Trueskill (Herbrich et al. 2007), which is a method for assigning a numerical skill to players in a player-vs-player game, given a win-loss record of games…”
Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “We attempt to only play games against reference agents that are nearby in skill in order to gain maximally useful information; we avoid playing agents more than 10 TrueSkill points away (corresponding to a winrate less than 15% or more than 85%).”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “To further evaluate the performance of different algorithms, we rank all the algorithms with the TrueSkill rating system (Herbrich, Minka, and Graepel, 2006), and the results are shown in the last row of Table 1 and Figure 3(b).”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “Metrics like ELO (Elo, 1978) or Trueskill (Herbrich et al., 2007) can more reliably measure whether performance is improving relative to previous policy versions or other policies in a population; however, these metrics still do not give insight into whether improved performance stems from new adapt”
A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning (2010, methodology): “It has since been adopted to many other two-player games such as Go and Scrabble, and, more recently, online computer gaming [Herbrich and Graepel, 2006].”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “The Elo rating system [230] was originally devised to rate chess player skills,30 TrueSkill [231] extended Elo by tracking uncertainty”
Skill Rating for Generative Models (2018, methodology): “Skill rating systems such as Elo [Elo, 1978] and TrueSkill [Herbrich et al., 2007] have been applied in the evaluation of game-playing systems [Silver et al., 2016, OpenAI, 2017], but to our knowledge ours is the ﬁrst application to generative models.”
Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence (2015, methodology): “The Thurstone (Case V) model has been used in a variety of both applied (Swets, 1973; Ross, 2007; Herbrich et al., 2007) and theoretical papers (Bramley, 2005; Krabbe, 2008; Nosofsky, 1985).”
Learning to Pass Expectation Propagation Messages (2013, methodology): “…we assume that models of interest are represented as factor graphs: the joint distribution over a set of random variables x = {x1, . . . , xD} is specified in terms of non-negative factors ψ1, . . . ,ψJ , which capture the relation between variables, and it decomposes as p(x) = 1/Z ∏J j=1 ψj(xψj ).”
R2: An Efficient MCMC Sampler for Probabilistic Programs (2014, background): “Our experimental results are also encouraging—we show that R2 is able to produce results of similar quality as state-of-theart probabilistic programming tools such as CHURCH and STAN with much shorter execution time.”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “The venerable ELO model (Glickman and Jones, 1999) and modern extensions (Herbrich et al., 2007) predict whether a player (subject) will defeat an opponent (item) with, again, a similar mathematical model.”
Streetscore -- Predicting the Perceived Safety of One Million Streetscapes (2014, methodology): “[22] to a ranked score using Trueskill [8].”
Bayesian reasoning and machine learning (2012, unknown): “3 Glicko and TrueSkill Glicko[114] and TrueSkill[130] are essentially Bayesian versions of the Elo model with the refinement that the latent ability is modelled, not by a single number, but by a Gaussian distribution p(αi|θi) = N ( αi μi, σ 2 i ) (22.”
Maximum Selection and Ranking under Noisy Comparisons (2017, unknown): “For example, Microsoft’s TrueSkill (Herbrich et al., 2006) software matches and ranks thousands of Xbox gamers based on individual game results.”
Stochastic Expectation Propagation (2015, background): “One approach is to swallow the large computational burden and simply use large data structures to store the approximating factors (e.g. TrueSkill [18]).”
Re-evaluating evaluation (2018, methodology): “Performance in AvA is often quantiﬁed using Elo ratings [16] or the closely related TrueSkill [17].”