Technical Brief: Agreement, the F-Measure, and Reliability in Information Retrieval

By George Hripcsak, A. Rothschild. J. Am. Medical Informatics Assoc., 2005

935 citations recorded by the source corpus.

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/269b88799c3d675cd603207a216bf2140787783f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Construction of an annotated corpus to support biomedical information extraction (2009, methodology): “We have thus chosen to follow [61] in choosing the F-Score to calculate IAA, as it can be applied straight-forwardly to all of the above annotation subtasks.”
Recent Advances in Clinical Natural Language Processing in Support of Semantic Analysis (2015, background): “Once a corpus is selected and a schema is defined, it is assessed for reliability and validity [9], traditionally through an annotation study in which annotators, e.g., domain experts and linguists, apply or annotate the schema on a corpus.”
BioInfer: a corpus for information extraction in the biomedical domain (2007, background): “• If may be difficult or impossible to calculate κ when the set of possible annotations is very large or not clearly defined [39].”
NCBI disease corpus: A resource for disease name recognition and concept normalization (2014, methodology): “We measured agreement by computing the F-measure between each pair of annotators that worked on the same set of documents, as also noted in [26,39].”
Applying the Naïve Bayes classifier with kernel density estimation to the prediction of protein-protein interaction sites (2010, neutral): “• F-measure (Hripcsak and Rothschild, 2005) combines precision and recall into their harmonic mean, and is defined as 2×(precision ×recall)/(precision + recall).”
Using clinical Natural Language Processing for health outcomes research: Overview and actionable suggestions for future advances (2018, neutral): “True negatives are rarely taken into consideration in NLP evaluation, often because this is intractable in text analysis [36].”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “Two psychologists took 12 psychology articles that were already published in prestigious psychology journals and resubmitted them with different author names and slight rewording of titles [151].”
SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations (2013, methodology): “…is released for the community to review 3See https://github.com/hllorens/TimeML-validator and improve.4 Inter-annotator agreement (measured with F1, as per Hripcsak and Rothschild (2005)) and the number of annotation passes per document were higher than in existing TimeML corpora, hence the name.”