Toward Optimal Active Learning through Sampling Estimation of Error Reduction

By Nicholas A. Roy, Andrew McCallum. 2001

1,072 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Meta & Continual Learning

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Another possible objective is reducing generalization error [109], which is estimated as follows: if the measure is log loss, then the entropy of the predicted class distribution is considered the error rate; if the measure is”
Automated Curriculum Learning for Neural Networks (2017, background): “We can take the self-prediction gain idea further and evaluate directly on the loss of interest, which has has also been considered in active learning (Roy and Mccallum, 2001).”
Bayesian Active Learning for Classification and Preference Learning (2011, methodology): “M L ] 24 D ec 2011 losses encountered after making decisions based on the data collected i.e. minimize the Bayes posterior risk [Roy and McCallum, 2001].”
What is Intrinsic Motivation? A Typology of Computational Approaches (2007, background): “…after an event e k has happened: r e t C H E t H E tk , ( , ) ( , )( ) = ⋅ − +( )1 (4) Examples of implementation of this information gain motivation can be found for instance in Fedorov (1972) and Roy and McCallum (2001) (but note that in these paper the term “motivation system” is not used).”
Trajectory Optimization using Reinforcement Learning for Map Exploration (2008, background): “Using naive Bayesian independence assumptions can often allow computationally efficient and accurate inference of the mean of a distribution, but the higherorder moments are generally biased and overconfident, leading to problems in active learning and exploration (Roy and McCallum 2001).”
Results of the Active Learning Challenge (2011, background): “Another more direct, but very computationally expensive approach is to tentatively add to the training set all possible candidate queries with one of the opposite labels and estimate how much generalization error reduction would result by adding it to the training set (Roy and McCallum, 2001).”
A machine learning approach to sentiment analysis in multilingual Web texts (2009, methodology): “…1994, Lewis and Gale 1994, Dagan and Engelson 1995, Freund et al. 1997, Liere and Tadepalli 1997, McCallum and Nigam 1998, Iyengar et al. 2000, Roy and McCallum 2001, Tong and Koller 2002, Baram et al. 2003, Saar-Tsechansky and Provost 2004, Osugi 2005) or a set of examples (Brinker 2003, Xu…”
Active Learning for Efficient Audio Annotation and Classification with a Large Amount of Unlabeled Data (2019, background): “While there are several different families of query strategies, including certaintybased sampling [20], query-by-committee [21], expected error reduction [22], and medoid-based AL [18], we focus on the certaintybased sampling strategy since there is a direct relation between “certainty” and the pred”
Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey (2009, methodology): “Two selective AL sampling methods were considered: Simple Margin (Tong and Koller, 2000) and Error Reduction (Roy and McCallum, 2001).”
Bootstrapping Intrinsically Motivated Learning with Human Demonstrations (neutral): “This inspired the creation of fully autonomous robots [17]–[22] with meta-exploration mechanisms monitoring the evolution of learning performances of the robot, in order to maximise informational gain, and with heuristics deﬁning the notion of interest [23]–[25].”
Semi-Supervised Active Learning for Sound Classification in Hybrid Learning Environments (2016, methodology): “This strategy has been adopted for text classification task with Naive Bayes models [25], and leads to a dramatic improvement over certainty-based and query-by-committee strategies.”
Ieee Transactions on Evolutionary Computation Intrinsic Motivation Systems for Autonomous Mental Development (neutral): “Moreover, there exists no efficient implementation for methods like optimal experiment design in continuous spaces, and already in discrete spaces the computational cost is high ([35]).”
Online Choice of Active Learning Algorithms (2003, methodology): “The second algorithm we discuss, proposed by (Roy & McCallum, 2001), is based on a different motivation: The algorithm chooses its next example to be labeled while attempting to reduce future generalization error probability.”
Active Learning Methods for Remote Sensing Image Classification (2009, methodology): “In [29], a multiclass algorithm is proposed.”
Cleaning Crowdsourced Labels Using Oracles For Statistical Classification (2018, background): “• Expected Error Reduction (ExpectError) [34] cleans the instance such that the current model’s error can be reduced the most, where the error is computed on noisy labels rather than estimated w.”