HEAD-QA: A Healthcare Dataset for Complex Reasoning

By David Vilares, Carlos Gómez-Rodríguez. 2019

151 citations recorded by the source corpus.

We present HEAD-QA, a multi-choice question answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. We then consider monolingual (Spanish) and cross-lingual (to English) experiments with information retrieval and neural techniques. We show that: (i) HEAD-QA challenges current methods, and (ii) the results lag well behind human performance, demonstrating its usefulness as a benchmark for future work.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Question Answering & Reading Comprehension, Medical & Health AI

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hannaneh Hajishirzi
2. Luke S. Zettlemoyer
3. Tushar Khot
4. Mohit Bansal
5. Jonathan Berant
6. Matt Gardner
7. Ashish Sabharwal
8. Kyle Richardson

In Evaluation & Benchmarks

1. Kyle Richardson
2. Tushar Khot
3. Luke S. Zettlemoyer
4. Oyvind Tafjord
5. Arman Cohan
6. Abhilasha Ravichander
7. Pradeep Dasigi
8. Yanai Elazar

In Question Answering & Reading Comprehension

1. Tushar Khot
2. Kyle Richardson
3. Luke S. Zettlemoyer
4. Jack Hessel
5. Abhilasha Ravichander
6. Oyvind Tafjord
7. Yanai Elazar
8. Ashish Sabharwal

In Medical & Health AI

1. Arman Cohan
2. Jack Hessel
3. Kyle Lo

Citations in the corpus

• GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, neutral): “, 2018), HeadQA (English) (Vilares and Gómez-Rodríguez, 2019), HellaSwag (Zellers et al.”
• OLMo: Accelerating the Science of Language Models (2024, background): “These tasks are headqa_en (Vilares and Gómez-Rodríguez, 2019), logiqa (Liu et al., 2020), mrpc (Dolan and Brock-ett, 2005), qnli (Wang et al., 2018), wic (Pilehvar and Camacho-Collados, 2018), and wnli (Wang et al., 2018).”
• A Comprehensive Overview of Large Language Models (2023, unknown): “Type Datasets/Benchmarks Multi-Task MMLU [242], SuperGLUE [3], BIG-bench [243], GLUE [244], BBH [243], CUGE [245], ZeroCLUE [246], FewCLUE [247], Blended Skill Talk [248], HELM [249], KLUE-STS [250] Language Understanding CoQA [251], WiC [252], Wikitext103 [253], PG19 [254], LCQMC [255], QQP [256], ”
• RWKV: Reinventing RNNs for the Transformer Era (2023, background): “• HeadQA (Vilares and Gómez-Rodríguez, 2019) A benchmark consisting of graduatelevel questions encompassing various fields such as medicine, nursing, biology, chemistry, psychology, and pharmacology.”