Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
By Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anya Belz, Aleksandar Savkov. 2022
66 citations recorded by the source corpus.
In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient’s clinical safety.To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced.
Subfields: Health Information Management, Molecular Biology, Public Health, Environmental and Occupational Health, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Molecular Biology
In Public Health, Environmental and Occupational Health
In Natural Language Processing
- Thibault Sellam
- Elizabeth Clark
- Sebastian Gehrmann
- Yejin Choi
- Yonatan Belinkov
- Tatsunori B. Hashimoto
- Greg Durrett
- Percy Liang
In Evaluation & Benchmarks
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “The same baseline was introduced for neural models on CNN/DM by Nallapati et al. (2017) who similarly find that it is extremely effective.”
- A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025, unknown): “21 reported 3.9 errors and 6.6 omissions per note as produced by a BART model and 1 error and 4 omissions per human-written note.”