Ratings and rankings: voodoo or science?

By Paolo Paruolo, Michaela Saisana, Andrea Saltelli. 2011

367 citations recorded by the source corpus.

Summary. Composite indicators aggregate a set of variables by using weights which are understood to reflect the variables’ importance in the index. We propose to measure the importance of a given variable within existing composite indicators via Karl Pearson's ‘correlation ratio’; we call this measure the ‘main effect’. Because socio‐economic variables are heteroscedastic and correlated, relative nominal weights are hardly ever found to match relative main effects; we propose to summarize their discrepancy with a divergence measure. We discuss to what extent the mapping from nominal weights to main effects can be inverted. This analysis is applied to six composite indicators, including the human development index and two popular league tables of university performance. It is found that in many cases the declared importance of single indicators and their main effect are very different, and that the data correlation structure often prevents developers from obtaining the stated importance, even when modifying the nominal weights in the set of non‐negative numbers with unit sum.

Subfields: Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/5a59b912be20127badacf800750c0811922886db

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

Citations in the corpus

Stochastic predictions of bulk properties of amorphous polyethylene based on molecular dynamics simulations (2014, neutral): “Quantitative measures can also be drawn, for example by measuring the correlation between Y and Xi, or even by estimating variance-based measures by nonlinear regression [24].”
Stochastic predictions of interfacial characteristic of polymeric nanocomposites (PNCs) (2014, background): “A scatter plot can draw the correlation between Y and Xi, or even by estimating variancebased measures by nonlinear regression [39].”
On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness (2018, background): “1 3 will soon find that this is far from actually happening in this aggregation setting, and this situation is the norm rather than the exception (Anderson and Zalinski 1988; Munda and Nardo 2005; Billaut et al. 2010; Rowley et al. 2012; Paruolo et al. 2013).”