Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.
By J. Ayers, Adam Poliak, Mark Dredze, E. Leas, Zechariah Zhu, Jessica B Kelley, D. Faix, A. Goodman, C. Longhurst, Michael Hogarth and 1 more. JAMA Internal Medicine, 2023
2,120 citations recorded by the source corpus.
Importance The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians. Objective To evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions. Design, Setting, and Participants In this cross-sectional study, a public and nonidentifiable database of questions from a public social media forum (Reddit's r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose "which response was better" and judged both "the quality of information provided" (very poor, poor, acceptable, good, or very good) and "the empathy or bedside manner provided" (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians. Results Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P < .001). Chatbot responses were rated of significantly higher quality than physician responses (t = 13.3; P < .001). The proportion of responses rated as good or very good quality (≥ 4), for instance, was higher for chatbot than physicians (chatbot: 78.5%, 95% CI, 72.3%-84.1%; physicians: 22.1%, 95% CI, 16.4%-28.2%;). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses (t = 18.9; P < .001). The proportion of responses rated empathetic or very empathetic (≥4) was higher for chatbot than for physicians (physicians: 4.6%, 95% CI, 2.1%-7.7%; chatbot: 45.1%, 95% CI, 38.5%-51.8%; physicians: 4.6%, 95% CI, 2.1%-7.7%). This amounted to 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot. Conclusions In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.
Subfields: Applied Psychology, Artificial Intelligence, Health Informatics
View Paper: https://www.semanticscholar.org/paper/a70c50bff01547f00ce341cd7e42797f11c4cdc7
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Vivek Natarajan
- Greg S. Corrado
- Jay J. Van Bavel
- Nenad Tomašev
- Ryutaro Tanno
- Jakob Nikolas Kather
- Shekoofeh Azizi
- Ilia Sucholutsky
Citations in the corpus
- HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, neutral): “…extensive human evaluation (Pfohl et al., 2024; Fleming et al., 2024; Tu et al., 2025; Tanno et al., 2025), comparison against expert baselines (Ayers et al., 2023; Goh et al., 2024; Tu et al., 2024, 2025; Singhal et al., 2025), or reflect realistic healthcare professional workflows (Dash et…”
- GPT is an effective tool for multilingual psychological text analysis (2024, background): “…GPT’s ability to help with computational social science tasks (49–54), detect misinformation (55), infer politicians’ ideologies (56), write persuasive political arguments (57), respond to patient questions (58), simulate human research participants (59–61), and model collective behavior (56).”
- Toward expert-level medical question answering with large language models (2025, neutral): “As LLMs become increasingly proficient at structured tests of knowledge, it is more important to delineate and assess their capabilities along clinically relevant dimensions 21,25 .”
- The future landscape of large language models in medicine (2023, background): “Through their text simpli fi cation capabilities 33 , LLMs may improve communication between healthcare staff and patients 34 .”
- Towards conversational diagnostic artificial intelligence (2025, positive): “This general finding is consistent with a prior study, where LLM responses were found to be more empa-thetic than the responses from clinicians to health questions posted on Reddit 24 .”
- RAG-Enhanced Open SLMs for Hypertension Management Chatbots (2025, unknown): “Although there is still room for further evaluation, the consistently high ratings reported in literature [ 6], suggest that we may consider empathy, in a first approximation, ensured by design if models are properly instructed.”