Reviewer bias in single- versus double-blind peer review
By Andrew Tomkins, M. Zhang, W. Heavlin. Proceedings of the National Academy of Sciences of the United States of America, 2017
496 citations recorded by the source corpus.
Significance Scientific peer review has been a cornerstone of the scientific method since the 1600s. Debate continues regarding the merits of single-blind review, in which anonymous reviewers know the authors of a paper and their affiliations, compared with double-blind review, in which this information is hidden. We present an experimental study of this question. In computer science, research often appears first or exclusively in peer-reviewed conferences rather than journals. Our study considers full-length submissions to the highly selective 2017 Web Search and Data Mining conference (15.6% acceptance rate). Each submission is simultaneously scored by two single-blind and two double-blind reviewers. Our analysis shows that single-blind reviewing confers a significant advantage to papers with famous authors and authors from high-prestige institutions. Peer review may be “single-blind,” in which reviewers are aware of the names and affiliations of paper authors, or “double-blind,” in which this information is hidden. Noting that computer science research often appears first or exclusively in peer-reviewed conferences rather than journals, we study these two reviewing models in the context of the 10th Association for Computing Machinery International Conference on Web Search and Data Mining, a highly selective venue (15.6% acceptance rate) in which expert committee members review full-length submissions for acceptance. We present a controlled experiment in which four committee members review each paper. Two of these four reviewers are drawn from a pool of committee members with access to author information; the other two are drawn from a disjoint pool without such access. This information asymmetry persists through the process of bidding for papers, reviewing papers, and entering scores. Reviewers in the single-blind condition typically bid for 22% fewer papers and preferentially bid for papers from top universities and companies. Once papers are allocated to reviewers, single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors, top universities, and top companies. The estimated odds multipliers are tangible, at 1.63, 1.58, and 2.10, respectively.
Subfields: Communication, Information Systems
View Paper: https://www.semanticscholar.org/paper/dfe9fe4e964ee45af15e392765d6babca4afd23e
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Communication
In Information Systems
Citations in the corpus
- Gender and international diversity improves equity in peer review (2018, result): “467 Our finding that the gender of the last authors is associated with a significant 468 difference in the rate at which full submissions were accepted at eLife stands in contrast 469 with a number of previous studies of journal peer review; these studies have found no 470 significant difference in ”
- Can We Automate Scientific Reviewing? (2021, background): “However, at the same time it is easy to identify many issues: expensiveness, slowness, existence of inconsistency (Langford and Guzdial, 2015) and bias (Tomkins et al., 2017), etc.”
- Men, women and STEM: Why the differences and what should be done? (2020, background): “• In computer science, conference papers that include female authors are just as likely to be accepted when the reviewers know the authors’ names (and thus potentially their sex) as when they don’t have this information (Tomkins et al., 2017).”