CliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension

By Simon Suster, Walter Daelemans. 2018

97 citations recorded by the source corpus.

We present a new dataset for machine comprehension in the medical domain. Our dataset uses clinical case reports with around 100,000 gap-filling queries about these cases. We apply several baselines and state-of-the-art neural readers to the dataset, and observe a considerable gap in performance (20% F1) between the best human and machine readers. We analyze the skills required for successful answering and show how reader performance varies depending on the applicable skills. We find that inferences using domain knowledge and object tracking are the most frequently required skills, and that recognizing omitted information and spatio-temporal reasoning are the most difficult for the machines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Question Answering & Reading Comprehension, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wenqiang Lei
2. Saku Sugawara
3. Tat-seng Chua
4. Akiko Aizawa
5. Satoshi Sekine
6. Soujanya Poria
7. Jianming Zheng
8. Ankit Pal

In Computer Vision and Pattern Recognition

1. Saku Sugawara
2. Tat-seng Chua
3. Akiko Aizawa
4. Satoshi Sekine
5. Soujanya Poria
6. Jianming Zheng
7. Ankit Pal
8. Chao Wang

In Evaluation & Benchmarks

1. Tat-seng Chua
2. Akiko Aizawa
3. Satoshi Sekine
4. Soujanya Poria
5. Jianming Zheng
6. Chao Wang
7. Saku Sugawara
8. Kentaro Inui

In Question Answering & Reading Comprehension

1. Tat-seng Chua
2. Akiko Aizawa
3. Satoshi Sekine
4. Soujanya Poria
5. Jianming Zheng
6. Chao Wang
7. Saku Sugawara
8. Kentaro Inui

In Medical & Health AI

1. Akiko Aizawa
2. Satoshi Sekine
3. Saku Sugawara
4. Kentaro Inui

Citations in the corpus

• What Makes Reading Comprehension Questions Easier? (2018, methodology): “…et al., 2015), Children’s Book Test (Hill et al., 2016), bAbI (Weston et al., 2015), WikiReading (Hewlett et al., 2016), LAMBADA (Paperno et al., 2016), Who-didWhat (Onishi et al., 2016), ProPara (Dalvi et al., 2018), MultiRC (Khashabi et al., 2018), CliCR (Suster and Daelemans, 2018), SQuAD (v2.”
• BioMRC: A Dataset for Biomedical Machine Reading Comprehension (2020, unknown): “It would be particularly interesting to explore if larger automatically generated datasets like BIOMRC and CLICR could be used to pre-train models, which could then be fine-tuned for human-generated QA or MRC datasets.”
• MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering (2022, background): “…(Yang et al., 2018; Lai et al., 2017; Zellers et al., 2018; Yagcioglu et al., 2018; Dua et al., 2019; Bajaj et al., 2018; Huang et al., 2019), extractive question answering (Hermann et al., 2015; Trischler et al., 2017), healthcare domain QA (Jin et al., 2019; ˇSuster and Daelemans, 2018; M¨oller”
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, background): “4 Boolean A rule text, a scenario and conversation history CliCR [134] Medical 104.”