An extensive review of tools for manual annotation of documents

By Mariana Neves, Jurica Seva. 2019

142 citations recorded by the source corpus.

Abstract Motivation Annotation tools are applied to build training and test corpora, which are essential for the development and evaluation of new natural language processing algorithms. Further, annotation tools are also used to extract new information for a particular use case. However, owing to the high number of existing annotation tools, finding the one that best fits particular needs is a demanding task that requires searching the scientific literature followed by installing and trying various tools. Methods We searched for annotation tools and selected a subset of them according to five requirements with which they should comply, such as being Web-based or supporting the definition of a schema. We installed the selected tools (when necessary), carried out hands-on experiments and evaluated them using 26 criteria that covered functional and technical aspects. We defined each criterion on three levels of matches and a score for the final evaluation of the tools. Results We evaluated 78 tools and selected the following 15 for a detailed evaluation: BioQRator, brat, Catma, Djangology, ezTag, FLAT, LightTag, MAT, MyMiner, PDFAnno, prodigy, tagtog, TextAE, WAT-SL and WebAnno. Full compliance with our 26 criteria ranged from only 9 up to 20 criteria, which demonstrated that some tools are comprehensive and mature enough to be used on most annotation projects. The highest score of 0.81 was obtained by WebAnno (of a maximum value of 1.0).

Subfields: Artificial Intelligence, Molecular Biology, Evaluation & Benchmarks, Software Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Alexander M. Rush
  2. Luke S. Zettlemoyer
  3. Colin Raffel
  4. Mohit Bansal
  5. Yejin Choi
  6. Jianfeng Gao
  7. Furu Wei
  8. Kyunghyun Cho

In Molecular Biology

  1. Tingwen Liu
  2. Quangang Li
  3. Colin Raffel
  4. Alexander M. Rush
  5. Stephen H. Bach
  6. Yanzeng Li
  7. Zhiqing Sun
  8. Yu Bowen

In Evaluation & Benchmarks

  1. Shanya Sharma Sharma
  2. Han Wang
  3. Manan Dey
  4. Albert Webson
  5. Tingwen Liu
  6. Quangang Li
  7. Colin Raffel
  8. Alexander M. Rush

In Software Engineering

  1. Tingwen Liu
  2. Quangang Li
  3. Yanzeng Li
  4. Yu Bowen

Citations in the corpus