Query by committee

By H. Sebastian Seung, Manfred Opper, Haim Sompolinsky. 1992

We propose an algorithm called query by commitee, in which a committee of students is trained on the same data set. The next query is chosen according to the principle of maximal disagreement. The algorithm is studied for two toy models: the high-low game and perceptron learning of another perceptron. As the number of queries goes to infinity, the committee algorithm yields asymptotically finite information gain. This leads to generalization error that decreases exponentially with the number of examples. This in marked contrast to learning from randomly chosen inputs, for which the information gain approaches zero and the generalization error decreases with a relatively slow inverse power law. We suggest that asymptotically finite information gain may be an important characteristic of good query algorithms.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “Query-by-Committee [104] extends uncertainty sampling by training a committee of models on the same labeled data.”
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (2019, methodology): “Speciﬁcally, we sample variables from the prior network as the random perturbation to estimate the model uncertainty following the idea of Query-By-Committee (Seung et al., 1992) and optimize model parameters by maximizing the ELBO.”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, background): “Ensemble methods (Seung et al., 1992; McCallumzy and Nigamy, 1998; Freund et al., 1997) generate queries that have the greatest disagreement between a set of classiﬁers.”
A Survey on Multi-view Learning (2013, background): “Active learning (Settles, 2009; Seung et al., 1992), sometimes called query”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “Early works have shown uncertainty sampling based active learning for logistic regression (Lewis and Gale, 1994) and neural networks (Seung et al., 1992; McCallum and Nigam, 1998) based on “query by committee”, an approach to estimate uncertainty by using an ensemble of neural networks.”
A Sequential Algorithm for Training Text Classifiers (1994, background): “Recently, several algorithms for learning via queries have been proposed that (cid:12)lter existing examples rather than creating arti(cid:12)cial ones [9, 10, 11].”
Bayesian Active Learning for Classification and Preference Learning (2011, methodology): “Perhaps the most ubiquitous is active learning for SVMs [Tong and Koller, 2001, Seung et al., 1992], where the volume of Version Space (VS) is used as a proxy for the posterior entropy.”
Batch mode active learning and its application to medical image classification (2006, methodology): “In several recent studies (Fine et al., 2002; Freund et al., 1997; Graepel & Herbrich, 2000; Seung et al., 1992), a number of distinct classification models are first generated.”
SLAC: A Sparsely Labeled Dataset for Action Classification and Localization (2017, unknown): “In active learning, this consensus method is known as query-by-committee (QBC) [25].”
Hyperparameter Learning for Graph Based Semi-supervised Learning Algorithms (2006, background): “One early and popular approach is called query-by-committee (QBC) (Freund et al., 1997; Seung et al., 1992), whereby an ensemble of classifiers are built and the examples with the highest classification variance are selected.”
ViewAL: Active Learning With Viewpoint Entropy for Semantic Segmentation (2019, background): “Yet another uncertainty-based approach is querying by committee where a committee of multiple models is trained on the labeled data, and unlabeled samples with least consensus are selected for labeling [45, 27].”
Semi-supervised learning by disagreement (2010, unknown): “Approaches of the former train a single learner and then query the unlabeled example on which the learner is least confident [45]; approaches of the latter generate multiple learners and then query the unlabeled example on which the learners disagree to the most [1] [63].”
Exploiting Unlabeled Data in Content-Based Image Retrieval (2004, methodology): “The latter methods such as [1][15] generate a committee of several learners and select the unlabeled instances on which the committee members disagree the most.”
Analysis of Perceptron-Based Active Learning (2009, background): “Although we do not further optimize our choice of the constant α, this choice itself may yield interesting future work, perhaps by allowing it to be a function of the dimension.”