Can We Automate Scientific Reviewing?

By Weizhe Yuan, Pengfei Liu, Graham Neubig. 2021

129 citations recorded by the source corpus.

The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question “can we automate scientific reviewing? ”, discussing the possibility of using natural language processing (NLP) models to generate peer reviews for scientific papers. Because it is non-trivial to define what a “good” review is in the first place, we first discuss possible evaluation metrics that could be used to judge success in this task. We then focus on the machine learning domain and collect a dataset of papers in the domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers as input and generate reviews as output. Comprehensive experimental results on the test set show that while system-generated reviews are comprehensive, touching upon more aspects of the paper than human-written reviews, the generated texts are less constructive and less factual than human-written reviews for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. Given these results, we pose eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research in this direction. We make relevant resource publicly available for use by future research: https://github. com/neulab/ReviewAdvisor. In addition, while our conclusion is that the technology is not yet ready for use in high-stakes review settings we provide a system demo, ReviewAdvisor (http://review.nlpedia.ai/), showing the current capabilities and failings of state-of-the-art NLP models at this task (see demo screenshot in A.2). A review of this paper written by the system proposed in this paper can be found in A.1.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pengfei Liu
2. Graham Neubig
3. Weizhe Yuan
4. Yixin Liu
5. Jinlan Fu
6. Xipeng Qiu
7. Mohit Bansal
8. Yang Liu

In Information Systems

1. Pengfei Liu
2. Graham Neubig
3. Weizhe Yuan

In Large Language Models

1. Weizhe Yuan
2. Pengfei Liu
3. Graham Neubig

In Natural Language Processing

1. Pengfei Liu
2. Graham Neubig
3. Weizhe Yuan
4. Jinlan Fu
5. Felix Stahlberg
6. Orhan Firat
7. Yixin Liu
8. Sebastian Ruder

Citations in the corpus

• BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “In this work, we operationalize our idea by using BART [32] as our backbone due to its superior performance in text generation [12,42,72].”
• How well do you know your summarization datasets? (2021, unknown): “The past few years have witnessed major break-throughs and improvements in automatic summarization (See et al., 2017; Celikyilmaz et al., 2018; Jadhav and Rajan, 2018; Liu and Lapata, 2019; Liu, 2019; Dou et al., 2020; Yuan et al., 2021; Liu et al., 2021).”