Predicting accuracy on large datasets from smaller pilot data

By Mark Johnson, Peter Anderson, Mark Dras, Mark Steedman. 2018

45 citations recorded by the source corpus.

Because obtaining training data is often the most difficult part of an NLP or ML project, we develop methods for predicting how much data is required to achieve a desired test accuracy by extrapolating results from models trained on a small pilot training dataset. We model how accuracy varies as a function of training size on subsets of the pilot data, and use that model to predict how much training data would be required to achieve the desired accuracy. We introduce a new performance extrapolation task to evaluate how well different extrapolations predict accuracy on larger training sets. We show that details of hyperparameter optimisation and the extrapolation models can have dramatic effects in a document classification task. We believe this is an important first step in developing methods for estimating the resources required to meet specific engineering performance targets.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, background): “A number of prior studies observed power-law relations between the generalization error and training data size (Cho et al., 2015; Miceli Barone et al., 2017; Johnson et al., 2018).”
Revisiting Neural Scaling Laws in Language and Vision (2022, methodology): “This has been used, for example, to estimate the required sample size in healthcare [9], neural machine translation (NMT) [19], and language models [23], among others [20, 22, 32].”