A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

By Dongyeop Kang, Bridger Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard H. Hovy, Roy Schwartz. 2018

282 citations recorded by the source corpus.

Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1),1 providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as ‘originality’ and ‘impact’.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Self-critiquing models for assisting human evaluators (2022, background): “[KAD18] introduce a dataset of critical peer reviews.”
Intrinsic Evaluation of Summarization Datasets (2020, methodology): “PeerRead (Kang et al., 2018) is a dataset of paper drafts from top-tier computer science venues as well as arXiv.”
What Can We Do to Improve Peer Review in NLP? (2020, methodology): “…revealed only four conference papers on peer review from a meta-research perspective: a paper-reviewer matching tool (Anjum et al., 2019), a corpus of reviews (Kang et al., 2018), and two experimental studies using NLP to explain the observed reviews (Caragea et al., 2019; Gao et al., 2019).”
FHI Oxford Technical Report # 2018-2 Predicting Human Deliberative Judgments with Machine Learning (2018, background): “These include datasets of movie reviews [47], reviewer scores for academic papers [48], and verdicts for legal cases [49].”
GORC: A large contextual citation graph of academic papers (2019, methodology): “Jeong et al. (2019) use a mod-iﬁed version of the PeerRead dataset (Kang et al., 2018) to generate contextual citations from full text for training an in-line citation recommendation model.”