Patient-facing chatbots and agents: what the evidence shows

These tools talk with patients directly: symptom checkers suggest possible conditions and how urgently to seek care, intake chatbots take a history before a visit, and voice agents call patients for health systems. FDA has said it intends to exercise enforcement discretion for symptom-checklist software that suggests possible conditions and when to see a clinician.

A 2022 systematic review of 10 studies found symptom checkers' primary-diagnosis accuracy ranged from 19% to 37.9% and triage accuracy from 48.8% to 90.1%. A five-year follow-up of 22 apps found median triage accuracy of 55.8% in 2020, close to 59.1% in 2015. In a randomized study of 1,298 members of the public, large language models alone identified the right conditions in 94.9% of scenarios, but participants using them identified relevant conditions in fewer than 34.5% of cases, no better than a control group. In a vignette test, OpenAI's consumer ChatGPT Health under-triaged 52% of gold-standard emergencies. Open questions include effects on where patients seek care and on their outcomes.