Targeting the Benchmark: On Methodology in Current Natural Language Processing Research
By David Schlangen. 2020
75 citations recorded by the source corpus.
It has become a common pattern in our field: One group introduces a language task, exemplified by a dataset, which they argue is challenging enough to serve as a benchmark. They also provide a baseline model for it, which then soon is improved upon by other groups. Often, research efforts then move on, and the pattern repeats itself. What is typically left implicit is the argumentation for why this constitutes progress, and progress towards what. In this paper, we try to step back for a moment from this pattern and work out possible argumentations and their parts.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Noah A. Smith
- Percy Liang
- Jianfeng Gao
- Yejin Choi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Quoc V. Le
- Trevor Darrell
- Andrew Zisserman
- Cordelia Schmid
- Ming-Hsuan Yang
- Xiaogang Wang
Citations in the corpus
- PaLM 2 Technical Report (2023, methodology): “This rubric evolved from literature review on evaluation (Rauh et al., 2022; Dev et al., 2021b; Bowman & Dahl, 2021; Sambasivan et al., 2021; Paullada et al., 2021; Rodriguez et al., 2021; Schlangen, 2020; Denton et al., 2020; Selbst et al., 2019; Jacobs & Wallach, 2021; Tomasev et al., 2021; Welty ”
- Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI (2020, methodology): “Motivated by categorization in [63], we set the “vulnerability boundary”—a hypothetical threshold for tasks without an associated undesirable event— right from natural language inference [7],”
- Underspecification in Scene Description-to-Depiction Tasks (2022, background): “As van Miltenburg (2020) argues, the dataset annotations which often act as extensional definitions of the task of interest (Schlangen, 2021) are often produced via under-specified crowdsourcing tasks that do not pay full attention to the rich space of possible text-image relationships described…”
- Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research (2021, methodology): “Datasets exemplify machine learning tasks, typically through a collection of input and output pairs 21 [1].”
- Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “, 2020) and may not reflect human language capabilities (Schlangen, 2020).”