Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models for Health Advice using Crowdsourced Clinical Cases

The proliferation of Large Language Models (LLMs) in high-stakes applications such as medical (self-)diagnosis and preliminary triage raises significant ethical and practical concerns about the effectiveness, appropriateness, and possible harmfulness of the use of these technologies for health-related concerns and queries. Some prior work has considered the effectiveness of LLMs in answering expert-written health queries/prompts, questions from medical examination banks, or questions from online forums. Unfortunately, none of these studies address the effectiveness of LLMs in answering everyday health concerns and queries typically asked by general users, which corresponds to the more prevalent use case for LLMs. To address this research gap, this paper presents the findings from a university-level competition that leveraged a novel, crowdsourced approach for evaluating the effectiveness and potential harmfulness of LLMs in answering everyday health queries. Over the course of a week, a total of 34 participants prompted four publicly accessible LLMs with 212 real (or imagined) health concerns, and the LLM generated responses were evaluated by a team of nine board-certified physicians. At a high level, our findings indicate that while ~76% of the 212 LLM responses were deemed to be accurate by physicians, nearly one-third (~33%) were flagged as potentially harmful. Further, with the help of medical professionals, we investigated whether retrieval augmented generation (RAG) variants of these LLMs (powered with a comprehensive medical knowledge base) can improve the quality of LLM responses. Finally, we derive qualitative insights to explain our quantitative findings by conducting interviews with seven medical professionals who were shown all the prompts in our competition. This paper aims to provide a more grounded understanding of how LLMs perform in real-world everyday health communication.