A sequential algorithm for training text classifiers

By David D. Lewis, William A. Gale. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Uncertainty sampling, for example, picks the instances that the partially trained model is least certain about (Lewis & Gale, 1994; Nigam et al., 2000), which could be interpreted as a measure of difficulty.”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “Some of our sampling and ﬁltering strategies based on entropy are inspired by uncertainty sampling, a standard active learning algorithm (Lewis and Gale, 1994).”
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, background): “Adversarial Annotation One recently adopted approach to constructing challenging datasets involves the use of an adversarial model to select examples that it does not perform well on, an approach which superficially is akin to active learning (Lewis and Gale, 1994).”
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping (2020, neutral): “Active learning techniques, which choose a data order using a criterion such as the model’s uncertainty (Lewis & Gale, 1994), have a rich history.”
Learning Algorithms for Active Learning (2017, methodology): “For instance, Lewis & Gale (1994) and Tong & Chang (2001) developed policies based on the conﬁdence of the classiﬁer, while Gilad-Bachrach et al. (2005) used the disagreement of a committee of classiﬁers.”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…Wang, Zhang, Li, Zhang, & Lin, 2016; Settles, 2009; Yoo & So Kweon, 2019; Wei, Iyer, & Bilmes, 2015), with the most common selection methods using uncertainty sampling (Lewis & Gale, 1994; Culotta & McCallum, 2005; Scheffer, Decomain, & Wrobel, 2001; Dagan & Engelson, 1995; Dasgupta & Hsu, 2008).”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “proaches use classifier confidence (Lewis and Gale, 1994), predicted expected error (Roy and McCallum, 2001), or model disagreement (Houlsby et al.”
A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning (2010, background): “This is called uncertainty sampling [Lewis and Gale, 1994].”
Visual Causal Feature Learning (2014, methodology): “The algorithm, inspired by the active learning techniques of uncertainty sampling (Lewis and Gale, 1994) and density weighing (Settles and Craven, 2008), starts off by training a causal neural network in Step 2.”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “MCD - classiﬁcation conﬁdence: query based on lowest softmax conﬁdence (Lewis and Gale, 1994).”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, neutral): “Active learning (Lewis and Gale, 1994), especially when using an uncertaintybased acquisition function, is also closely related to DADC.”
Scaling to Very Very Large Corpora for Natural Language Disambiguation (2001, background): “A sample is presumed to be more useful for training the more uncertain its classification label is. Uncertainty can be judged by the relative weights assigned to different labels by a single classifier ( Lewis and Gale , 1994 ).”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “…are also reminiscent of active learning methods (Settles, 2009; Peris and Casacuberta, 2018; P.V.S and Meyer, 2019), such as uncertainty sampling (Lewis and Gale, 1994) which selects (unlabeled) data points, which a model trained on a small labeled subset, has least conﬁdence in, or predicts as…”
Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration (2016, background): “, uncertainty sampling (Lewis and Gale 1994; Settles 2009), query by committee (Seung, Opper, and Sompolinsky 1992), expected model change (Settles, Craven, and Ray 2008), expected error reduction (Zhu, Lafferty, and Ghahramani 2003), expected variance reduction (Zhang and Oles 2000)).”
Active Learning with Sampling by Uncertainty and Density for Word Sense Disambiguation and Text Classification (2008, background): “…natural language processing (NLP) tasks, such as word sense disambiguation (Chen et al., 2006; Zhu and Hovy, 2007), text classification (TC) (Lewis and Gale, 1994; McCallum and Nigam, 1998), named entity recognition (NER) (Shen et al., 2004), chunking (Ngai and Yarowsky, 2000), information…”
An instance level analysis of data complexity (2014, methodology): “For example, uncertainty sampling (Lewis and Gale, 1994) selects an unlabeled instance x whose labeling the learning algorithm is least certain about:”
Adversarial Active Learning for Deep Networks: a Margin Based Approach (2018, background): “For example, we empirically noticed in our experiments that uncertainty selection, or uncertainty sampling (Lewis & Gale, 1994), may perform worse than passive random selection.”
Active Learning Literature Survey (2009, methodology): “Perhaps the simplest and most commonly used query framework is uncertainty sampling (Lewis and Gale, 1994).”