Measuring Data
By Margaret Mitchell, Alexandra Sasha Luccioni, Nathan Lambert, M. Gerchick, Angelina McMillan-Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, Douwe Kiela. 2022
21 citations recorded by the source corpus.
We identify the task of measuring data to quantitatively characterize the composition of machine learning data and datasets. Similar to an object's height, width, and volume, data measurements quantify different attributes of data along common dimensions that support comparison. Several lines of research have proposed what we refer to as measurements, with differing terminology; we bring some of this work together, particularly in fields of computer vision and language, and build from it to motivate measuring data as a critical component of responsible AI development. Measuring data aids in systematically building and analyzing machine learning (ML) data towards specific goals and gaining better control of what modern ML systems will learn. We conclude with a discussion of the many avenues of future work, the limitations of data measurements, and how to leverage these measurement approaches in research and practice.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Siva Reddy
- Luke S. Zettlemoyer
- Mohit Bansal
- Yonatan Belinkov
- Jianfeng Gao
- Thomas Wolf
- Anna Rogers
- Daniel Fried
In Computer Vision and Pattern Recognition
In Evaluation & Benchmarks
- Aleksandra Piktus
- Qian Liu
- Anna Rogers
- Siva Reddy
- Nicholas Meade
- Harm de Vries
- Manan Dey
- Niklas Muennighoff
Citations in the corpus
- StarCoder: may the source be with you! (2023, background): “This pressing need for understanding data (Mitchell et al., 2022) is being increasingly recognized and operationalized in the form of dataset inspection tools and toolkits (Akiki et al., 2023; Marone & Van Durme, 2023; Piktus et al., 2023).”
- The ROOTS Search Tool: Data Transparency for LLMs (2023, background): “In an encouraging new trend, we see researchers exploring ways to quantitatively describe large datasets (Mitchell et al., 2022).”
- MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, unknown): “However, many such corpora are released without proper documentation [45].”