An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition

By G. Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, M. Zschunke, M. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, D. Polychronopoulos and 12 more. 2015

831 citations recorded by the source corpus.

This article provides an overview of the first BioASQ challenge, a competition on large-scale biomedical semantic indexing and question answering (QA), which took place between March and September 2013. BioASQ assesses the ability of systems to semantically index very large numbers of biomedical scientific articles, and to return concise and user-understandable answers to given natural language questions by combining information from biomedical articles and ontologies. The 2013 BioASQ competition comprised two tasks, Task 1a and Task 1b. In Task 1a participants were asked to automatically annotate new PubMed documents with MeSH headings. Twelve teams participated in Task 1a, with a total of 46 system runs submitted, and one of the teams performing consistently better than the MTI indexer used by NLM to suggest MeSH headings to curators. Task 1b used benchmark datasets containing 29 development and 282 test English questions, along with gold standard (reference) answers, prepared by a team of biomedical experts from around Europe and participants had to automatically produce answers. Three teams participated in Task 1b, with 11 system runs. The BioASQ infrastructure, including benchmark datasets, evaluation mechanisms, and the results of the participants and baseline methods, is publicly available. A publicly available evaluation infrastructure for biomedical semantic indexing and QA has been developed, which includes benchmark datasets, and can be used to evaluate systems that: assign MeSH headings to published articles or to English questions; retrieve relevant RDF triples from ontologies, relevant articles and snippets from PubMed Central; produce “exact” and paragraph-sized “ideal” answers (summaries). The results of the systems that participated in the 2013 BioASQ competition are promising. In Task 1a one of the systems performed consistently better from the NLM’s MTI indexer. In Task 1b the systems received high scores in the manual evaluation of the “ideal” answers; hence, they produced high quality summaries as answers. Overall, BioASQ helped obtain a unified view of how techniques from text classification, semantic indexing, document and passage retrieval, question answering, and text summarization can be combined to allow biomedical experts to obtain concise, user-understandable answers to questions reflecting their real information needs.

Subfields: Ecology, Pharmacology, Sociology and Political Science, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Pharmacology

In Sociology and Political Science

In Evaluation & Benchmarks

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “e-based medicine. Beyond TREC, another notable effort is the BioASQ workshop series, which has been organizing evaluation challenges on biomedical semantic indexing and question answering since 2013 [Tsatsaronis et al., 2015]. Among the earliest known 102Who doesn’t want to help cure cancer? 103Just”
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, methodology): “To reduce the length to meet our second requirement ( ≤ 800 to-kens), for each question we manually selected a TREC (Baudiˇs and ˇSediv´y, 2015) The Text REtrieval Conference (TREC) dataset is curated from the TREC QA tasks (Voorhees and Tice, 2000) from 1999-2002.”
PubMedQA: A Dataset for Biomedical Research Question Answering (2019, methodology): “Yes/No QA: Datasets such as HotpotQA (Yang et al., 2018), Natural Questions (Kwiatkowski et al., 2019), ShARC (Saeidi et al., 2018) and BioASQ (Tsatsaronis et al., 2015) contain yes/no questions as well as other types of questions.”
emrQA: A Large Corpus for Question Answering on Electronic Medical Records (2018, background): “…other hand, complex domainspecific MC datasets such as MCTest (Richardson et al., 2013), biological process modeling (Berant et al., 2014), BioASQ (Tsatsaronis et al., 2015), InsuranceQA (Feng et al., 2015), etc have been limited in scale (500-10K) because of the complexity of the task or the…”
Neural Domain Adaptation for Biomedical Question Answering (2017, background): “BioASQ The BioASQ challenge provides a biomedical QA dataset (Tsatsaronis et al., 2015) consisting of questions, relevant contexts (called snippets) from PubMed abstracts and possible answers to the question.”
Large language models encode clinical knowledge (2022, background): “We plan to expand the benchmark to other relevant datasets, such as those probing question answering ability from electronic medical records [65] or those requiring pre-clinical biomedical knowledge [83], in future work.”
AUEB at BioASQ 7: Document and Snippet Retrieval (2019, background): “bioasq [26] is a biomedical document classification, retrieval, and question answering competition.”
Zero-shot Neural Retrieval via Domain-targeted Synthetic Query Generation (2020, methodology): “BIOASQ dataset contains English biomedical questions collected from the document ranking task of BIOASQ competition (Tsatsaronis et al., 2015).”