An extensive review of tools for manual annotation of documents
By Mariana Neves, Jurica Seva. 2019
142 citations recorded by the source corpus.
Abstract Motivation Annotation tools are applied to build training and test corpora, which are essential for the development and evaluation of new natural language processing algorithms. Further, annotation tools are also used to extract new information for a particular use case. However, owing to the high number of existing annotation tools, finding the one that best fits particular needs is a demanding task that requires searching the scientific literature followed by installing and trying various tools. Methods We searched for annotation tools and selected a subset of them according to five requirements with which they should comply, such as being Web-based or supporting the definition of a schema. We installed the selected tools (when necessary), carried out hands-on experiments and evaluated them using 26 criteria that covered functional and technical aspects. We defined each criterion on three levels of matches and a score for the final evaluation of the tools. Results We evaluated 78 tools and selected the following 15 for a detailed evaluation: BioQRator, brat, Catma, Djangology, ezTag, FLAT, LightTag, MAT, MyMiner, PDFAnno, prodigy, tagtog, TextAE, WAT-SL and WebAnno. Full compliance with our 26 criteria ranged from only 9 up to 20 criteria, which demonstrated that some tools are comprehensive and mature enough to be used on most annotation projects. The highest score of 0.81 was obtained by WebAnno (of a maximum value of 1.0).
Subfields: Artificial Intelligence, Molecular Biology, Evaluation & Benchmarks, Software Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Alexander M. Rush
- Luke S. Zettlemoyer
- Colin Raffel
- Mohit Bansal
- Yejin Choi
- Jianfeng Gao
- Furu Wei
- Kyunghyun Cho
In Molecular Biology
- Tingwen Liu
- Quangang Li
- Colin Raffel
- Alexander M. Rush
- Stephen H. Bach
- Yanzeng Li
- Zhiqing Sun
- Yu Bowen
In Evaluation & Benchmarks
- Shanya Sharma Sharma
- Han Wang
- Manan Dey
- Albert Webson
- Tingwen Liu
- Quangang Li
- Colin Raffel
- Alexander M. Rush
In Software Engineering
Citations in the corpus
- PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, background): “Systems for Annotating Data Most work on collecting annotations has focused on labels and other annotations at the level of individual examples (Neves and Ševa, 2021).”
- FITAnnotator: A Flexible and Intelligent Text Annotation System (2021, methodology): “In recent years, the NLP community has developed several annotation tools (Neves and Ševa, 2019).”