An Analysis of Active Learning Strategies for Sequence Labeling Tasks
By Burr Settles, Mark W. Craven. 2008
1,169 citations recorded by the source corpus.
Active learning is well-suited to many problems in natural language processing, where unlabeled data may be abundant but annotation is slow and expensive. This paper aims to shed light on the best active learning approaches for sequence labeling tasks such as information extraction and document segmentation. We survey previously used query selection strategies for sequence models, and propose several novel algorithms to address their shortcomings. We also conduct a large-scale empirical comparison using multiple corpora, which demonstrates that our proposed methods advance the state of the art.
Subfields: Artificial Intelligence, Natural Language Processing, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Percy Liang
- Graham Neubig
- Jianfeng Gao
- Trevor Darrell
- Luke S. Zettlemoyer
- Quoc V. Le
In Natural Language Processing
- Graham Neubig
- Lucia Specia
- Trevor Cohn
- Percy Liang
- Luke S. Zettlemoyer
- Mohit Bansal
- Andrew McCallum
- Jianfeng Gao
In Meta & Continual Learning
- Burr Settles
- Fei Sha
- Raymond J. Mooney
- Trevor Cohn
- Mario Fritz
- Jaime G. Carbonell
- Bernt Schiele
- Daniel Beck
Citations in the corpus
- Rewriting a Deep Generative Model (2020, background): “Interactive machine learning systems aim to improve training through human interaction in labeling [15,22,68], or by allowing a user to to aid in the model optimization process via interactive feature selection [19,27,64,50] or model and hyperparameter selection [39,61,37].”
- A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, methodology): “Densityweighting [107] is a way to improve the above techniques by choosing instan-”
- Visual Causal Feature Learning (2014, methodology): “The algorithm, inspired by the active learning techniques of uncertainty sampling (Lewis and Gale, 1994) and density weighing (Settles and Craven, 2008), starts off by training a causal neural network in Step 2.”
- A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “…active learning have rapidly identified the challenges of robust application and pitfalls faced through the use of heuristics (Roy and McCallum, 2001; Settles and Craven, 2008; Li and Guo, 2013), the latter are nonetheless once again dominant in the era of deep learning (Beluch et al., 2018;…”
- Active Learning by Labeling Features (2009, methodology): “We use random (rand) selection, uncertainty sampling (US) (using sequence entropy, normalized by sequence length) and information density (ID) (Settles and Craven, 2008) to select instance queries.”
- Learning Active Learning from Data (2017, methodology): “Among AL methods, some cater to specific classifiers, such as those relying on Gaussian processes [16], or to specific applications, such as natural language processing [32, 25], sequence labeling tasks [28], visual recognition [21, 18], semantic segmentation [33], foreground-background segmentation”
- Active Learning Literature Survey (2009, methodology): “Alternatively, Set-tles and Craven (2008) approximate the matrix with its diagonal vector, which can be inverted in only O ( K ) time.”
- Adaptive Active Learning for Image Classification (2013, methodology): “In [18, 26], the most uncertain instance is taken as the one that has the largest entropy on the conditional distribution over its labels.”
- Learning to Actively Learn Neural Machine Translation (2018, background): “• Total Token Entropy (TTE) We sort monolingual sentences based on their TTE which has been shown to be a strong AL heuristic (Settles and Craven, 2008) for”
- Semi-Supervised Active Learning for Sequence Labeling (2009, methodology): “Different approaches to AL have been successfully applied to a wide range of NLP tasks (Engelson and Dagan, 1996; Ngai and Yarowsky, 2000; Tomanek et al., 2007; Settles and Craven, 2008).”
- Learning how to Active Learn: A Deep Reinforcement Learning Approach (2017, methodology): “Uncertainty sampling we use the total token entropy measure (Settles and Craven, 2008), which takes the instance x maximising ∑|x| t=1H(yt|x, φ), where H is the token entropy.”
- Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis (2011, methodology): “When adapting sequence-based prediction methods, most active learning approaches have focused on picking full sentences that are valuable to annotate (Ringger et al., 2007; Settles and Craven, 2008).”
- Machine Learning Paradigms for Speech Recognition: An Overview (2013, background): “Recent work proposed to select inputs not only based on an uncertainty/disagreement measure but also on a “density” measure [171], [176].”
- Reducing Annotation Effort for Quality Estimation via Active Learning (2013, methodology): “We experiment with two of them: Uncertainty Sampling (US) (Lewis and Gale, 1994) and Information Density (ID) (Settles and Craven, 2008).”
- Uncertainty Estimation for Language Reward Models (2022, background): “While simple, this has been found to be a strong baseline [38, 49, 14].”
- Robust Active Label Correction (2018, methodology): “For selecting noisy examples for correction, we adopt the strategy from active learning to select those points that promise to change the model the most in expectation (Settles and Craven, 2008; Settles et al., 2008).”
- Results of the Active Learning Challenge (2011, background): “Since discriminative probabilistic models are usually trained with gradient-based optimization, the “change” imparted can be measured by the magnitude of the gradient (Settles et al., 2008a).”