emrQA: A Large Corpus for Question Answering on Electronic Medical Records

By Anusri Pampari, Preethi Raghavan, Jennifer J. Liang, Jian Peng. 2018

264 citations recorded by the source corpus.

We propose a novel methodology to generate domain-specific large-scale question answering (QA) datasets by re-purposing existing annotations for other NLP tasks. We demonstrate an instance of this methodology in generating a large-scale QA dataset for electronic medical records by leveraging existing expert annotations on clinical notes for various NLP tasks from the community shared i2b2 datasets. The resulting corpus (emrQA) has 1 million questions-logical form and 400,000+ question-answer evidence pairs. We characterize the dataset and explore its learning potential by training baseline models for question to logical form and question to answer mapping.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “Similar facets are explored in medical domain datasets (Pampari et al., 2018; Šuster and Daelemans, 2018) which contain automatically generated queries on medical records based on predefined templates.”
PubMedQA: A Dataset for Biomedical Research Question Answering (2019, background): “Some works proposed automatically constructed biomedical QA datasets (Pampari et al., 2018; Pappas et al., 2018; Kim et al., 2018), which have much larger sizes.”
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks (2020, background): “emrQA (Pampari et al. 2018)) or simulated worlds (Labutov et al.”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “This has led Smith to call the medical journals and pharmaceutical companies “uneasy bedfellows” and advocate that medical journals not publish such studies but instead serve as a forum to review their findings and limitations [260].”
Deep learning in clinical natural language processing: a methodical review (2019, result): “For instance, despite the fact that the studies had already been published, our ACL Anthol-ogy search missed both emrQA 116 and CliCR 117 whereas a current search with the exact same criteria on the ACL Anthology would have returned those studies.”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, background): “Specifically, existing work has presented datasets for the medical domain (Tsatsa-ronis et al., 2015; Pampari et al., 2018; Jin et al., 2019, 2021; Pal et al., 2022), legal (Guha et al., 2023), technology (Dos Santos et al., 2015) and others.”
BioMRC: A Dataset for Biomedical Machine Reading Comprehension (2020, methodology): “Several biomedical MRC datasets exist, but have orders of magnitude fewer questions than BIOMRC (Ben Abacha and Demner-Fushman, 2019) or are not suitable for a cloze-style MRC task (Pampari et al., 2018; Ben Abacha et al., 2019; Zhang et al., 2018).”
Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (2021, methodology): “Answering dataset emrQA (Pampari et al., 2018) and the Overnight dataset (Wang et al.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “This instability is more matters for Healthcare than other general applications.”
Cross-modal Memory Networks for Radiology Report Generation (2022, methodology): “Recently, many advanced NLP techniques (e.g., pre-trained language models) have been applied to tasks in the medical domain (Pampari et al., 2018; Zhang et al., 2018; Wang et al., 2018; Alsentzer et al., 2019; Tian et al., 2019, 2020a; Wang et al., 2020; Lee et al., 2020; Song et al., 2020).”
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021, background): “Some examples include emrQA (Pampari et al., 2018), BioRead (Pappas et al.”
A large language model for electronic health records (2022, methodology): “We used the emrQA dataset 44 , which is widely used as a benchmark dataset for MQA.”
When does pretraining help?: assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings (2021, background): “[23] posit that whole-word masking may be most suitable for domain adaptation on emrQA [33], a corpus for question answering on electronic medical records, because most words in emrQA are tokenized to sub-word WordPiece tokens [48] in base BERT due to the high frequency of unique, domain-specific me”