Does the “surprisingly popular” method yield accurate crowdsourced predictions?

By Abraham M. Rutchick, Bryan J. Ross, Dustin P. Calvillo, Catherine C. Mesick. 2020

13 citations recorded by the source corpus.

The “surprisingly popular” method (SP) of aggregating individual judgments has shown promise in overcoming a weakness of other crowdsourcing methods—situations in which the majority is incorrect. This method relies on participants’ estimates of other participants’ judgments; when an option is chosen more often than the average metacognitive judgments of that option, it is “surprisingly popular” and is selected by the method. Although SP has been shown to improve group decision making about factual propositions (e.g., state capitals), its application to future outcomes has been limited. In three preregistered studies, we compared SP to other methods of aggregating individual predictions about future events. Study 1 examined predictions of football games, Study 2 examined predictions of the 2018 US midterm elections, and Study 3 examined predictions of basketball games. When applied to judgments made by objectively assessed experts, SP performed slightly better than other aggregation methods. Although there is still more to learn about the conditions under which SP is effective, it shows promise as a means of crowdsourcing predictions of future outcomes.

Subfields: Economics and Econometrics, General Decision Sciences, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

Samuel R. Bowman

In Management Science and Operations Research

In Evaluation & Benchmarks

Citations in the corpus

What Do NLP Researchers Believe? Results of the NLP Community Metasurvey (2022, methodology): “this method to predictions of the future are positive, though more mixed (Lee et al., 2018; Rutchick et al., 2020).”