Active Learning by Labeling Features

By Gregory Druck, Burr Settles, Andrew McCallum. 2009

186 citations recorded by the source corpus.

Methods that learn from prior information about input features such as generalized expectation (GE) have been used to train accurate models with very little effort. In this paper, we propose an active learning approach in which the machine solicits "labels" on features rather than instances. In both simulated and real user experiments on two sequence labeling tasks we show that our active learning method outperforms passive learning with features as well as traditional active learning with instances. Preliminary experiments suggest that novel interfaces which intelligently solicit labels on multiple features facilitate more efficient annotation.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Training Classifiers with Natural Language Explanations (2018, background): “Previous works have relied on identifying relevant parts of the input such as labeling features (Druck et al., 2009; Raghavan et al., 2005; Liang et al., 2009), highlighting rationale phrases in Both cohorts showed signs of optic nerve toxicity due to ethambutol.”
Annotator rationales for visual recognition (2011, background): “Work in natural language processing (NLP) explores whether humans can pick out words relevant for a given document category as a form of human feature selection [6, 18, 28].”
Leveraging the Wisdom of the Crowd for Fine-Grained Recognition (2016, background): “More importantly, there are much fewer discriminative features compared to categorization at the basic level.”
Active Learning Literature Survey (2009, background): “4) can lead to reduced annotation costs for human oracles (Raghavan et al., 2006; Druck et al., 2009; Vijayanarasimhan and Grauman, 2009a).”
From Theories to Queries (2011, background): “…reason about annotation cost, several authors have found that alternative query types (such as labeling features rather than instances, see Section 2.4) can lead to reduced annotation costs for human oracles (Raghavan et al., 2006; Druck et al., 2009; Vijayanarasimhan and Grauman, 2009a).”
Explanatory Interactive Machine Learning (2019, background): “More recently, researchers explored feature supervision [1, 10, 11, 23, 24, 30] and rationales [32, 40, 41], which leverage both labeland feature-level (or sentence-level, for rationales) supervision with the aim of improving learning efficiency.”
Toward interactive training and evaluation (2011, methodology): “As in previous work [6, 7, 16], in this paper we predominantly use input feature label distribution constraints.”
Toward an Architecture for Never-Ending Language Learning (2010, unknown): “Previous work has shown that labeling features (e.g., context patterns) rather than instances can lead to significant improvements in terms of reducing human annotation time (Druck, Settles, and McCallum 2009).”
Closing the Loop: Fast, Interactive Semi-Supervised Annotation With Queries on Features and Instances (2011, background): “…not assigned to classes, but instead ﬂagged for “relevance” to the task (Godbole et al., 2004; Raghavan et al., 2006), or (2) feature queries are posed just like instance queries: a word is presented to the annotator, who must choose among the labels (Druck et al., 2009; Attenberg et al., 2010).”
Semi-automated screening of biomedical citations for systematic reviews (2010, background): “We are also interested in the emerging work on active learning over features rather than instances [37], which may be helpful in identifying phrases or UMLS concepts particularly characteristic of relevant citations.”
Fine-Grained Crowdsourcing for Fine-Grained Recognition (2013, unknown): “by asking humans to directly provide annotation rationales [9], to label features in NLP tasks [10], to describe the differences between pairs of images [20], or to perform tasks that are parts of the machine pipeline [23].”
Group-Based Active Learning of Classification Models (2017, background): “For example, (Druck, Settles, and McCallum 2009) solicits information on NLP features rather than words (instances).”