Active Learning Literature Survey

By Burr Settles. 2009

6,701 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

ALICE: Active Learning with Contrastive Natural Language Explanations (2020, background): “learning is that, if the learning algorithm is allowed to choose the data from which it learns, it will perform better than randomly selecting training samples (Settles, 2009).”
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop (2015, background): “ﬁer on that subset, asking the classiﬁer to predict labels and conﬁdences, and then selecting a small candidate set for further consideration. In many ways, this process is similar to active learning [14, 18, 3, 20]. However, there is a key difference: we aim to get correct labels for all examples i”
Refining Language Models with Compositional Explanations (2021, background): “active learning [39]) or to determine which model to use [9].”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, methodology): “Active learning approaches have been used to accelerate annotation (Tsuruoka et al., 2008), although this typically assumes access to a pool or stream of unlabelled data for which the learning algorithm can query labels (Settles, 2009).”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “When such tests involve ML model evaluation, this takes on the flavor of active learning [130].”
Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classification (2017, neutral): “Active Learning is a branch of ML that focuses on cost-efﬁcient ways to engage experts in a way to improve future iterations of a model, 25 and this body of work can provide tools for data scientists to not only address discrimination but also mitigate classiﬁcation errors that have a high human…”
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition (2015, background): “There are many possible criterion for active learning [42].”
Physics-Informed Neural Networks for Cardiac Activation Mapping (2020, background): “A common approach is to select the location where the uncertainty is the highest, which we can quantify by evaluating the entropy of the posterior distribution p(T|D) [21].”
Learning Algorithms for Active Learning (2017, methodology): “708.00088v1 [cs.LG] 31 Jul 2017 Learning Algorithms for Active Learning 2. Related Work Various heuristics have been proposed to guide the selection of which examples to label during active learning (Settles, 2010). For instance,Lewis & Gale(1994) andTong & Chang(2001) developed policies bas”
Automated Curriculum Learning for Neural Networks (2017, background): “Another related area is the ﬁeld of active learning, where similar gain signals have been used to guide decisions about which data point to label next (Settles, 2010).”
Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “Active learning algorithms can be considered as organized and constrained self-exploration processes [19, 20, 21, 22, 9].”
Selective Experience Replay for Lifelong Learning (2018, methodology): “…and can be interpreted as the surprise of a particular transition This idea is similar to the idea of uncertainty sampling in active learning (Settles 2010), identifying unexpected inputs in history compression (Schmidhuber 1993), and it has been effectively used as the foundation for a…”
Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…problem is also closely related to active learning strategies (Cohn, Atlas, & Ladner, 1994; Lin & Parikh, 2017; Wang, Zhang, Li, Zhang, & Lin, 2016; Settles, 2009; Yoo & So Kweon, 2019; Wei, Iyer, & Bilmes, 2015), with the most common selection methods using uncertainty sampling (Lewis & Gale,…”
Machine Unlearning (2019, background): “This selection is a challenging problem, akin to the issues that underlie active learning [48].”
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, background): “This preference is consistent with common variance reduction strategies in active learning [43].”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “Active learning is often employed in domains where data is cheap but labeling is expensive, and is motivated by the idea that not all data points are equally valuable when it comes to learning (Settles, 2009; Dasgupta, 2004).”
Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks (2017, background): “Well-calibrated uncertainties would be very useful in areas such as Bayesian optimization [54] or active learning [53], as well as when using predictions from machine learning algorithms in the real world and noticing when models are in domains away from those upon which they were trained.”