AlpacaTag: An Active Learning-based Crowd Annotation Framework for Sequence Tagging

By Bill Yuchen Lin, Dong-Ho Lee, Frank F. Xu, Ouyu Lan, Xiang Ren. 2019

38 citations recorded by the source corpus.

We introduce an open-source web-based data annotation framework (AlpacaTag) for sequence tagging tasks such as named-entity recognition (NER). The distinctive advantages of AlpacaTag are three-fold. 1) Active intelligent recommendation: dynamically suggesting annotations and sampling the most informative unlabeled instances with a back-end active learned model; 2) Automatic crowd consolidation: enhancing real-time inter-annotator agreement by merging inconsistent labels from multiple annotators; 3) Real-time model deployment: users can deploy their models in downstream systems while new annotations are being made. AlpacaTag is a comprehensive solution for sequence labeling tasks, ranging from rapid tagging with recommendations powered by active learning and auto-consolidation of crowd annotations to real-time model deployment.

Subfields: Artificial Intelligence, Computer Science Applications, Evaluation & Benchmarks, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiang Ren
2. Bill Yuchen Lin
3. Mohit Bansal
4. Luke S. Zettlemoyer
5. Yejin Choi
6. Alexander M. Rush
7. Colin Raffel
8. Hannaneh Hajishirzi

In Computer Science Applications

1. Xiang Ren
2. Alexander M. Rush
3. Stephen H. Bach
4. Victor Sanh

In Evaluation & Benchmarks

1. Bill Yuchen Lin
2. Xiang Ren
3. Shanya Sharma Sharma
4. Han Wang
5. Leonardo Neves
6. Qinyuan Ye
7. Manan Dey
8. Rahul Khanna

In Information Extraction

1. Colin Raffel
2. Alexander M. Rush

Citations in the corpus

• PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, methodology): “More recently, many annotation systems have begun to incorporate learned models to improve workflow, using techniques such as ac-S1 tive learning (Lin et al., 2019; Li et al., 2021) and example recommendation (Lee et al., 2020; Kiela et al., 2021).”
• LEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanation (2020, methodology): “To resolve this issue, AlpacaTag, an annotation framework for sequence labeling (Lin et al., 2019) attempts to provide annotation recommendations from a learned sequence labeling model that is incrementally updated by batches of incoming human annotations.”