Reducing Labeling Effort for Structured Prediction Tasks

By Aron Culotta, Andrew McCallum. 2005

466 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…Wang, Zhang, Li, Zhang, & Lin, 2016; Settles, 2009; Yoo & So Kweon, 2019; Wei, Iyer, & Bilmes, 2015), with the most common selection methods using uncertainty sampling (Lewis & Gale, 1994; Culotta & McCallum, 2005; Scheffer, Decomain, & Wrobel, 2001; Dagan & Engelson, 1995; Dasgupta & Hsu, 2008).”
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (2019, background): “To find new concepts, the heuristic uncertainty estimation methods (Tong and Koller, 2001; Culotta and McCallum, 2005) in active learning (Balcan et al.”
SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup (2020, background): “So far, various uncertainty-based (Scheffer et al., 2001; Culotta and McCallum, 2005; Kim et al., 2006) and committee-based approaches (Dagan and Engelson, 1995) 2017) further improve the aforementioned active learning approaches to improve the sampling diversity as well as the model’s…”
A Framework and Tool for Collaborative Extraction of Reliable Information (2013, background): “, 2000), or by correcting mistakes made by automatic extraction methods (Kristjansson et al., 2004; Culotta and McCallum, 2005).”
Deep Active Learning for Named Entity Recognition (2017, methodology): “Least Confidence (LC): This method sorts examples in descending order by the probability of not predicting the most confident sequence from the current model (Lewis and Gale, 1994; Culotta and McCallum, 2005):”
Using “Annotator Rationales” to Improve Machine Learning for Text Categorization (2007, background): “In active learning, the annotator classifies only documents where the system so far is less confident (Lewis and Gale, 1994), or in an information extraction setting, incrementally corrects details of the system’s less confident entity segmentations and labelings (Culotta and McCallum, 2005).”
Accelerating the annotation of sparse named entities by dynamic sentence selection (2008, methodology): “Here we present some experimental results to show that the annotation cost for each sentence could be reduced if we take advantage of the N-best sequences output from the CRF tagger by using the idea presented in [16].”
Active Learning Literature Survey (2009, methodology): “This sort of strategy has been shown to work well, for example, with conditional random ﬁelds or CRFs (Lafferty et al., 2001) for active learning in information extraction tasks (Culotta and McCallum, 2005; Settles and Craven, 2008).”
From Theories to Queries (2011, methodology): “One proposed approach for reducing annotation effort in active learning involves using the current trained model to assist in the annotation of queries by pre-labeling them in structured learning tasks like parsing (Baldridge and Osborne, 2004) or information extraction (Culotta and McCallum, 2005).”
An Analysis of Active Learning Strategies for Sequence Labeling Tasks (2008, methodology): “…on the conventions of uncertainty sampling , where the learner queries the instance about which it has the least certainty (Scheffer et al., 2001; Culotta and McCallum, 2005; Kim et al., 2006), or query-by-committee , where a “committee” of models selects the instance about which its members…”
Extracting Commonsense Properties from Embeddings with Limited Human Guidance (2018, methodology): “We adopt the conﬁdence measure ( Culotta and McCallum, 2005) to access the uncertainty of a given example: Good candidates for acquisition should have high uncertainty measure, but we also want to avoid querying outliers.”