A Sequential Algorithm for Training Text Classifiers

By David D. Lewis, William A. Gale. 1994

The ability to cheaply train text classifiers is critical to their use in information retrieval, content analysis, natural language processing, and other tasks involving data which is partly or fully textual. An algorithm for sequential sampling during machine learning of statistical classifiers was developed and tested on a newswire text categorization task. This method, which we call uncertainty sampling, reduced by as much as 500-fold the amount of training data that would have to be manually classified to achieve a given level of effectiveness.

Subfields: Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Optimization & Training

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Uncertainty sampling, for example, picks the instances that the partially trained model is least certain about (Lewis & Gale, 1994; Nigam et al., 2000), which could be interpreted as a measure of difficulty.”
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping (2020, neutral): “Active learning techniques, which choose a data order using a criterion such as the model’s uncertainty (Lewis & Gale, 1994), have a rich history.”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Uncertain Examples Uncertainty Sampling [78] is the simplest in active learning and chooses the next unlabeled example that the model prediction is most uncertain.”
Learning Algorithms for Active Learning (2017, methodology): “For instance, Lewis & Gale (1994) and Tong & Chang (2001) developed policies based on the conﬁdence of the classiﬁer, while Gilad-Bachrach et al. (2005) used the disagreement of a committee of classiﬁers.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “Other approaches use classiﬁer conﬁdence (Lewis and Gale, 1994), predicted expected error (Roy and McCallum, 2001), or model disagreement (Houlsby et al., 2011).”
Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration (2016, background): “Various query strategies have been proposed to choose the instances to be labeled (e.g., uncertainty sampling (Lewis and Gale 1994; Settles 2009), query by committee (Seung, Opper, and Som-polinsky 1992), expected model change (Settles, Craven, and Ray 2008), expected error reduction (Zhu,…”
A Survey on Malware Detection Using Data Mining Techniques (2017, background): “…the data scarcity problem, optimize the learning beneﬁt from domain experts’ feedback, and reduce the cost of acquiring labeled examples for supervised learning, has been intensively studied in recent years [Lewis and Gale 1994; Muslea et al. 2006; Seung et al. 1992; Nguyen and Smeulders 2004].”
Dual Strategy Active Learning (2007, background): “Uncertainty sampling [3] selects the example on which the learner has lowest classiﬁcation certainty.”
Selection Via Proxy: Efficient Data Selection For Deep Learning (2019, methodology): “For core-set selection, we try three methods to identify a subset of points: uncertainty sampling with entropy [Lewis and Gale, 1994, Settles, 2012], facility location [Wolf, 2011], and forgetting events [Toneva et al., 2019].”
Selective Annotation Makes Language Models Better Few-Shot Learners (2022, background): “• Least-conﬁdence (Lewis & Gale, 1994) iteratively adds least-conﬁdent examples to the annotated set.”
Results of the Active Learning Challenge (2011, background): “Perhaps the simplest and most commonly used query strategy is uncertainty sampling (Lewis and Gale, 1994).”
Unsupervised Activity Recognition Using Automatically Mined Common Sense (2005, background): “Active learning (Lewis & Gale 1994) and automated feature selection algorithms (Guyon & Elisseeff 2003) can focus labeling on the most profitable instances.”
Active and Semisupervised Learning for the Classification of Remote Sensing Images (2014, background): “One of the ﬁrst strategies introduced in the literature is based on uncertainty sampling [32], which aims at selecting the closest sample to the decision boundary.”