What Can We Do to Improve Peer Review in NLP?

By Anna Rogers, Isabelle Augenstein. 2020

62 citations recorded by the source corpus.

Peer review is our best tool for judging the quality of conference submissions, but it is becoming increasingly spurious. We argue that a part of the problem is that the reviewers and area chairs face a poorly defined task forcing apples-to-oranges comparisons. There are several potential ways forward, but the key difficulty is creating the incentives and mechanisms for their consistent implementation in the NLP community.

Subfields: Artificial Intelligence, Communication, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “As Rogers and Augenstein (2020) point out, resource papers are already underappreciated and increasing what counts as acceptable documentation for a resource paper may lead to fewer such papers being written.”
Can We Automate Scientific Reviewing? (2021, background): “Although peer review has been adopted by most journals and conferences to identify important and relevant research, its effectiveness is being continuously questioned (Smith, 2006; Langford and Guzdial, 2015; Tomkins et al., 2017; Gao et al., 2019; Rogers and Augenstein, 2020).”
“This is a Problem, Don’t You Agree?” Framing and Bias in Human Evaluation for Natural Language Generation (2020, background): “During the peer review process reviewers may default to heuristics to simplify the task of review, including rejecting papers where models do not achieve SOTA results (Rogers and Augenstein, 2020).”
When Combating Hype, Proceed with Caution (2021, background): “…et al., 2021; Bowman and Dahl, 2021; Nan-gia et al., 2021; Gehrmann et al., 2021) and research into better statistical reporting and publication norms (Dodge et al., 2019; Card et al., 2020; Rogers and Augenstein, 2020; van Miltenburg et al., 2021) seem especially high-impact under this lens.”
Changing the World by Changing the Data (2021, background): “Most task-speciﬁc tracks (question answering, summarization, dialogue etc.) are supposed to receive both engineering and data submissions, but in that setting the interdisciplinary tension may lead to resource papers voted down simply for being resource papers (Rogers and Augenstein, 2020).”