Metrology for AI: From Benchmarks to Instruments

By Chris Welty, Praveen K. Paritosh, Lora Aroyo. 2019

32 citations recorded by the source corpus.

In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations. We begin with the intuitive observation that evaluating the performance of an AI system is a form of measurement. In all other science and engineering disciplines, the devices used to measure are called instruments, and all measurements are recorded with respect to the characteristics of the instruments used. One does not report mass, speed, or length, for example, of a studied object without disclosing the precision (measurement variance) and resolution (smallest detectable change) of the instrument used. It is extremely common in the AI literature to compare the performance of two systems by using a crowd-sourced dataset as an instrument, but failing to report if the performance difference lies within the capability of that instrument to measure. To illustrate the adoption of metrology to benchmark datasets we use the word similarity benchmark WS353 and several previously published experiments that use it for evaluation.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

PaLM 2 Technical Report (2023, methodology): “This rubric evolved from literature review on evaluation (Rauh et al., 2022; Dev et al., 2021b; Bowman & Dahl, 2021; Sambasivan et al., 2021; Paullada et al., 2021; Rodriguez et al., 2021; Schlangen, 2020; Denton et al., 2020; Selbst et al., 2019; Jacobs & Wallach, 2021; Tomasev et al., 2021; Welty ”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “Welty et al. (2019) advocate for the more precise reporting of the focus and abilities of test sets and metrics in ML broadly, with a focus on issues surrounding statistical power.”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, methodology): “For a broader discussion on best practices for measuring AI systems, we refer the reader to the existing studies, such as Welty et al. (2019); van der Lee et al. (2019).”
When Combating Hype, Proceed with Caution (2021, background): “Welty et al. (2019), Linzen (2020), Ribeiro et al. (2020), Raji et al. (2021), Bowman and Dahl (2021), and Dehghani et al. (2021), among many others, discuss the challenges involved in designing evaluations that yield trustworthy and accurate depictions of the capabilities of ML models.”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “Often playing the role of commons [49], benchmark datasets exhibit “comedy of the commons” whereby usage of a dataset increases its value, due to the comparisons enabled by its role as a measuring instrument [144], while the patient and careful work of caring for datasets (including the collection, ”