Scaling to Very Very Large Corpora for Natural Language Disambiguation

By Michele Banko, Eric Brill. 2001

801 citations recorded by the source corpus.

The amount of readily available on-line text has reached hundreds of billions of words and continues to grow. Yet for most core natural language tasks, algorithms continue to be optimized, tested and compared after training on corpora consisting of only one million words or less. In this paper, we evaluate the performance of different learning methods on a prototypical natural language disambiguation task, confusion set disambiguation, when trained on orders of magnitude more labeled data than has previously been used. We are fortunate that for this particular application, correctly labeled training data is free. Since this will often not be the case, we examine methods for effectively exploiting very large corpora when labeled data comes at a cost.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Scaling Laws for Neural Language Models (2020, background): “Some early [BB01, Goo01] work found power-law scalings between performance and dataset size.”
Deep Learning Scaling is Predictable, Empirically (2017, methodology): “The Banko and Brill (2001) confusion set disambiguation work claims that the model size required to ﬁt the data grows "log-linearly".”
Beyond human-level accuracy: computational challenges in deep learning (2019, background): “The DL community commonly accepts that model accuracy improves as training dataset size grows (e.g., [6, 7, 36]).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “date(Albert Einstein, 1879/03/14); these are referred to as “tuples” or extracted facts. Relations usually draw from a relatively constrained 11A well-known observation dating back decades, at least [Banko and Brill, 2001]. 12Note that the term “distant supervision” was coined in the early 2000s, so”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “While the conventional wisdom in supervised learning is that more training instances usually helps (Banko and Brill, 2001; Sun et al., 2017; Hestness et al., 2017), in our setup, the model’s performance saturates when only 64 instances per task are used for training.”
Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks (2008, background): “Since the performance of many natural language processing tasks is limited by the amount and quality of data available to them (Banko and Brill, 2001), one promising alternative for some tasks is the collection of non-expert annotations.”
Embracing data abundance: BookTest Dataset for Reading Comprehension (2016, background): “…generating cloze-style question-answering datasets; 2. it is widely accepted that more data can significantly improve performance of most models (Banko and Brill, 2001; Halevy et al., 2009); 3. training a model takes only about two hours on the CBT or about two days on the Daily Mail dataset,…”
Deep neural network based malware detection using two dimensional binary program features (2015, background): “This makes static analysis very amenable for machine learning approaches, which tends to perform better as data size increases [5].”
ECG-based heartbeat classification for arrhythmia detection: A survey (2016, positive): “Researchers from the machine learning community have shown that the size/diversity of the databases used for the construction of methods impacts more than the choice of the learning algorithm and/or employed techniques [164].”
Referential translation machines for predicting semantic similarity (2015, background): “Active learning selects a subset of the training set that will benefit a learning algorithm the most (Banko and Brill 2001) without using the test set, and transductive learning makes use of test instances, which can sometimes be accessible at training time, to learn specific models tailored towards”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, background): “Larger training datasets support better performance and generalization (Banko and Brill, 2001; Halevy et al., 2009; Sun et al., 2017), especially for deep models.”
Bootstrapping Coreference Classifiers with Multiple Machine Learning Algorithms (2003, methodology): “…performance of the Blum and Mitchell cotraining algorithm with that of two existing singleview bootstrapping algorithms — self-training with bagging (Banko and Brill, 2001) and EM (Nigam et al., 2000) — on coreference resolution, and show that single-view weakly supervised learners are a viable…”
Using the Web to Obtain Frequencies for Unseen Bigrams (2003, background): “Banko and Brill (2001a; 2001b) experiment with context-sensitive spelling correction, a task for which large amounts of data can be obtained straightforwardly, as no manual annotation is required.”
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition (2008, background): “Other fields, such as speech, routinely use 10(6) data points for training, since they have found that large training sets are vital for achieving low er rors rates in testing [2].”
Text: now in 2D! A framework for lexical expansion with contextual similarity (2013, result): “Comparing our proposal with existing measures, we feel that the effectiveness of semantic similarity measures on large corpora has been reconfirmed: on more data, simpler measures perform as well or even better than measures that were intended to give good results for small collections – an insight ”
Introduction to the Special Issue on the Web as Corpus (2003, background): “Another argument is made vividly by Banko and Brill (2001).”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “While the conventional wisdom in supervised learning is that more training instances usually helps (Banko and Brill, 2001; Sun et al., 2017; Hestness et al., 2017), in our setup of cross-task generalization, the model’s performance saturates when only 64 instances per task are used (7)That is, linea”
Neural network analysis of sleep stages enables efficient diagnosis of narcolepsy (2017, background): “The success of machine learning depends on the size and quality of the data on which the model is trained and evaluated 51,52 .”