Mitigating the Paucity-of-Data Problem: Exploring the Effect of Training Corpus Size on Classifier Performance for Natural Language Processing

By Michele Banko, Eric Brill. 2001

116 citations recorded by the source corpus.

In this paper, we discuss experiments applying machine learning techniques to the task of confusion set disambiguation, using three orders of magnitude more training data than has previously been used for any disambiguation-in-string-context problem. In an attempt to determine when current learning methods will cease to benefit from additional training data, we analyze residual errors made by learners when issues of sparse data have been significantly mitigated. Finally, in the context of our results, we discuss possible directions for the empirical natural language research community.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Constructive Prediction of the Generalization Error Across Scales (2019, background): “Data scaling: It has long been recognized that more data improves performance, and various studies report such trends in both computer vision (e.g., Zhu et al., 2012; Sun et al., 2017) and language processing tasks (e.g., Banko & Brill, 2001; Talmor & Berant, 2019).”
Scaling to Very Very Large Corpora for Natural Language Disambiguation (2001, background): “Surely we cannot reasonably expect that the manual annotation of one billion words along with corresponding parse trees will occur any time soon (but see (Banko and Brill 2001) for a discussion that this might not be completely infeasible).”
Web-Scale Distributional Similarity and Entity Set Expansion (2009, background): “Banko and Brill (2001) show that Web-scale data helps with confusion set disambiguation while Lapata and Keller (2005) find that the Web is a good source of n-gram counts for unsupervised models.”
Best practices for convolutional neural networks applied to visual document analysis (2003, result): “This conclusion is supported by evidence from other application areas, such as text [ 8 ].”
Biases in Predicting the Human Language Model (2014, background): “Researchers in computational linguistics often assume that more data is always better than less data (Banko and Brill, 2001).”
Predicting accuracy on large datasets from smaller pilot data (2018, methodology): “Unfortunately our field only offers fairly impractical advice, e.g., that more data increases accuracy (Banko and Brill, 2001); we currently have no practical methods for estimating how much data or what quality of data is required to achieve a target accuracy goal.”
Detecting offensive tweets via topical feature discovery over a large scale twitter corpus (2012, background): “…and Subject Descriptors I.2.7 [Artificial Intelligence]: Natural Language Processing—Text analysis; I.5.2 [Pattern Recognition]: Design Methodology—Classifier design and evaluation General Terms Algorithms, Languages, Human Factors Keywords Twitter, Hadoop, topic modeling, machine learning”