Who grades the answers
American clinicians asked one AI reference tool 271 million questions in two and a half years. How often such tools get the answer right, on the questions clinicians actually ask, remains largely unmeasured.
Listen to this special topic
read by an AI voiceYou know the question. It comes at the end of a shift, about a drug interaction or a dose in renal failure, and it used to go to your colleague standing in the corridor. Now it often goes to a machine. Between January 2024 and June 2026, clinicians in the United States put 271 million of those questions to a single piece of software. A cross-sectional study of its query logs, published in JAMA on Oct. 15, counted 952,008 clinicians with verified provider numbers on OpenEvidence, which answers clinical questions and shows its citations.
The study did not measure whether a single one of those answers was right.
That is the curbside consult at industrial scale. How these tools perform on the treatment and medication questions clinicians actually put to them “remains largely unmeasured,” the study's authors write, which limits any clinician's ability to judge reliability where the tools get used most. That is not a research question. It is an audit. It belongs to the company, because a product asked this many questions should be graded on the questions it is asked, by clinicians it does not pay.
Independent evaluation is scarce, in the Nature Medicine team's word, and its verdicts are mixed. The team's own is unflattering. Writing in June, a team led from NYU Langone Health tested OpenEvidence and Wolters Kluwer's UpToDate Expert AI against three general-purpose models from OpenAI, Google and Anthropic, using two public benchmarks, one of them exam-style questions, and 100 real queries physicians had typed at one health system. Twelve clinicians reviewed the answers blind. That produced 1,800 ratings. The general-purpose models won all three comparisons, and on the real queries the clinical tools did about as well as Google's automatic search summaries. These were answers scored by clinicians, not care delivered to patients.
OpenEvidence rejected the finding. Becker's Hospital Review reported that in a June 15 letter to the journal the company asked for a retraction, a public apology and an independent review. The company argued the paper used flawed methods and that the public benchmarks may have sat in the models' training data. The letter has not been published, this publication has not seen it, and this account rests on that reporting. On the benchmark point the paper's authors agree with them. The models “may have been exposed to MedQA or HealthBench during training,” they write, and they treat the blinded review of real queries, free of that problem, as their primary evidence. By the same account the senior author said the team stands by the paper, and the journal pointed to its process for formal rebuttals without addressing the retraction demand directly. In September the journal published a critique of the paper, by researchers at the University of Chicago and the University of California, San Diego, with a reply from its authors.
The company has a study on its side, by academic authors who report no affiliation with it. A preprint it helped design, whose survey it administered and whose physicians it paid, in which 149 physicians rated answers to 620 real questions from its platform and preferred its own by 25 to 39 percentage points. It has not been peer reviewed. Among the other evaluations the company pointed to, by Becker's account, was a peer-reviewed study of five patients. Mayo Clinic physicians put questions to the tool from five retrospective cases, and four independent physicians rated the answers accurate and evidence-based in all five. Five cases cannot tell anyone how often a tool is right across 271 million answers.
Where the tested models go wrong is not random either. In Annals of Internal Medicine this month, researchers in Israel built 225 scenarios from 15 Choosing Wisely recommendations, the specialty societies' own lists of low-value tests and treatments, and put them to six commercial models for 4,050 answers. The models called for the discouraged test or treatment in 5.4% of answers and withheld an appropriate one in 0.6%. Most of that came from two of the 15 recommendations. On the larger one, a small thyroid nodule found by accident, more than two in five answers called for an ultrasound, where the society guidance, the authors note, is openly contested.
Doctors who reach for the tool a dozen times a shift will say these tests measure the wrong thing. The objection deserves a hearing. The Annals paper calls itself a benchmark rather than real-world use, built on single prompts and scored by another model with physicians adjudicating. The Nature Medicine queries came from physicians using a general-purpose chatbot. And the honest comparison is not a perfect library. It is what the question would otherwise have got, which is often a half-remembered guideline and a colleague's guess. Every one of those points holds. Not one of them argues against measuring. Many of the tasks used to evaluate clinical AI “may not match the tasks for which physicians actually use it,” the usage study concludes. The company holds both the questions and the answers.
So publish the audit only the company can run. Take a random sample of real answers. Report how often the answer is correct, and how often the source beneath it supports the claim. Have clinicians it neither employs nor pays do the judging. Release the method so somebody else can repeat it. The publishers licensing their journals to it hold a lever, and the JAMA Network is among them. An audit is a fair price for lending a journal's authority to a machine's answer. Until then, read the company's own terms, which say it plainly. “OpenEvidence Content is not intended to provide medical advice, diagnosis or treatment.” A curbside consult generally shifts no responsibility to the colleague who gives it, and the citation under each answer is there to be opened.
Counting the questions was the easy part. Grading the answers is the part that would tell a doctor something, and the study that did the counting says that work remains largely undone.
- What the JAMA study counted, and who wrote itPatel and colleagues, "Clinician Use of an Artificial Intelligence-Enabled Clinical Information Platform," JAMA, published online Oct. 15, 2026 (doi:10.1001/jama.2026.21868). A cross-sectional study of user query logs from all OpenEvidence accounts with a verified National Provider Identifier, Jan. 1, 2024, through June 30, 2026: 952,008 clinicians, including 556,200 physicians, and 271 million queries. A stratified random sample of 38.8 million queries was classified by a large language model; treatment and management accounted for 33.1% of physician queries and diagnosis for 13.9%. Monthly active users rose from 355 in January 2024 to 703,265 in June 2026. Three authors are listed with the affiliation "OpenEvidence, Miami, Florida." The disclosures state that Dr Finlayson "reported being a salaried employee of OpenEvidence, which provided data and contributed to study analysis"; that Dr Ray "reported receiving personal fees from OpenEvidence as a full-time employee during the conduct of the study"; and that Dr Zack "reported being the chief medical officer of OpenEvidence during the conduct of the study." The author contributions line reads "Obtained funding: Finlayson."Source: JAMA (Patel et al.)
- The study's stated limitsThe paper's limitations include that "data from a single AI-enabled clinical information platform were analyzed"; that "query classification relied on an LLM rather than human annotation, which may have introduced misclassification"; and that the authors "could not determine whether queries occurred during patient encounters." Its discussion says performance of these tools on treatment selection, medication guidance and evidence synthesis "remains largely unmeasured, limiting clinicians' ability to assess reliability where these tools are most often applied," and that "many of the tasks used to evaluate clinical AI may not match the tasks for which physicians actually use it." The accompanying editor's note, "Engaging With Medical Evidence in the Age of AI" (JAMA, Oct. 15, 2026; doi:10.1001/jama.2026.21894), states that the study did not evaluate the quality of generated responses, clinicians' assessments of accuracy or usefulness, or effects on patient care, and carries the disclosure that "The JAMA Network is among the publishers that license journal content to OpenEvidence for display in response to queries."Sources: JAMA (Patel et al.), JAMA editor's note
- An independent head-to-head evaluationVishwanath and colleagues, "General-purpose large language models outperform specialized clinical AI tools on medical benchmarks," Nature Medicine, June 12, 2026, 32(7):2405-2409 (doi 10.1038/s41591-026-04431-5), led from NYU Langone Health; full text read in the PubMed Central record (PMC13375586). Three stages: 500 MedQA questions, 500 HealthBench items, and a real clinical query benchmark of 100 de-identified queries put by physicians to a general-purpose language model, with 12 US clinicians performing blinded review that produced 1,800 model-question annotations. GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6 outperformed OpenEvidence and UpToDate Expert AI in all three evaluations, and on the real-query benchmark the authors found the clinical tools comparable to auto-enabled Google Search AI Overview. The paper reports MedQA accuracy of 97.4% for Gemini (95% CI, 95.6% to 98.5%), 89.6% for OpenEvidence (86.6% to 92.0%) and 88.4% for UpToDate (85.3% to 90.9%), and states that none of the models produced more harmful content or hallucinations than any of the others. Its authors note that models "may have been exposed to MedQA or HealthBench during training" and treat the blinded real-query review as primary evidence.Source: Nature Medicine (DOI)
- OpenEvidence's response, and the journal'sThe letter is not published, and this publication has not seen it. The only account of it that could be opened is Becker's Hospital Review, June 16, 2026, and by that report, in a June 15 letter OpenEvidence asked Nature Medicine for a retraction, a public apology and an independent review, arguing that the paper relies on flawed methods. By the same report the company said the public benchmarks may have been in the models' training data, and pointed to other evaluations, including a Mayo Clinic study it said found its tool accurate and guideline-concordant. Also by that report: senior author Eric Oermann, MD, said the authors continue to stand by the study and its results; Nature Medicine did not address the retraction demand directly, and its chief editor, João Monteiro, MD, PhD, said the journal welcomes scientific debate and pointed to its Matters Arising process; and Peter Bonis, MD, of Wolters Kluwer said the study confused clinical quality with complete-sounding answers. Those statements are paraphrased here from that report. On Sept. 3, 2026, Nature Medicine published a critique titled Limited benchmarks constrain the conclusions of a general-purpose versus clinical AI comparison, by Brett Beaulieu-Jones of the University of Chicago and Shamim Nemati of the University of California, San Diego (32(10):3595-3597; doi 10.1038/s41591-026-04638-6), with a reply from Vishwanath, Aphinyanaphongs and Oermann (32(10):3598-3599; doi 10.1038/s41591-026-04637-7). Both were read in their PubMed records only.Source: Becker's Hospital Review
- The preprint the company helped designBecker's Hospital Review, July 13, 2026, on a preprint posted to arXiv on June 27, 2026: 149 practicing physicians across 36 states rated AI answers to 620 real point-of-care questions taken from OpenEvidence's platform, plus 187 HealthBench questions, judging accuracy, clinical utility, source quality, verifiability and completeness. OpenEvidence posted positive win margins on all five dimensions, from 25 to 39 percentage points, over Claude Opus 4.8, Gemini 3.1 Pro and GPT-5.5. Becker's reported that the authors are based at UC San Francisco, Harvard Medical School, Stanford and other institutions and report no affiliation with OpenEvidence, and that "OpenEvidence codesigned the data collection plan, administered the survey and paid participating physicians." The preprint has not been peer reviewed.Source: Becker's Hospital Review
- The Choosing Wisely benchmarkDean Ariel, Stav Endelman and Guy Laban, "Overuse and False Restraint in Large Language Model Recommendations for Choosing Wisely Scenarios," Annals of Internal Medicine (doi 10.7326/ANNALS-26-01667; PubMed publication date Oct. 20, 2026; reported by AuntMinnie on Oct. 19, 2026); abstract read in the PubMed record. A benchmark evaluation: 15 Choosing Wisely recommendations were used to develop 225 scenarios, 10 per recommendation in which restraint was appropriate and 5 in which action was appropriate. Six commercial large language models, three runs each, generated 4,050 responses using a clinician-perspective prompt, through the production programming interfaces of OpenAI, Anthropic and Google from February to March 2026. The abstract does not name the six models individually. Overuse was 5.4% (95% CI, 0.6% to 12.2%) and absent for 8 of 15 recommendations; it reached 42.8% for thyroid ultrasound for an incidental subcentimeter nodule and 28.3% for routine electrocardiography in low-risk adults. False restraint was 0.6% (CI, 0.1% to 1.2%). Stated limits: single prompts without follow-up, a benchmark rather than real-world use, and responses classified by an LLM judge with physician adjudication. Funding: none.Sources: Annals of Internal Medicine (DOI), AuntMinnie
- The five-case Mayo Clinic studyRyan T. Hurt and colleagues, "The Use of an Artificial Intelligence Platform OpenEvidence to Augment Clinical Decision-Making for Primary Care Physicians," Journal of Primary Care and Community Health, April 16, 2025, 16:21501319251332215 (doi 10.1177/21501319251332215); abstract and full text read in the PubMed and PubMed Central records. Five patient cases were analyzed retrospectively across hypertension, hyperlipidemia, type 2 diabetes mellitus, depression and obesity; physicians posed specific clinical questions and four independent physicians rated the responses on a 0 to 4 scale. Mean scores were clarity 3.55, relevance 3.75, support 3.35 and satisfaction 3.60, while impact on clinical decision-making was 1.95; that item asked how far the tool's recommendation had modified or reinforced the treating physician's original plan, and the tool "primarily reinforced rather than modified plans." The authors conclude that "prospective trials are needed to evaluate its utility in complex cases and multidisciplinary settings."
- What the company saysIn a PR Newswire release dated July 10, 2026, OpenEvidence announced EvidenceGrade, which it says "grades and visualizes, in real time, the quality of the published evidence" cited in and used to inform every answer by "a novel method that builds on the GRADE framework," and describes its platform as "sourced, cited, and grounded in peer-reviewed medical literature." Samuel Finlayson, MD, PhD, its senior vice president of medical AI, said that "Not all evidence is equally certain," and that the feature is "a complement to traditional evidence synthesis." The release says the majority of US physicians use OpenEvidence daily. The platform's Terms of Use, last updated May 6, 2025, say "OpenEvidence Content is not intended to provide medical advice, diagnosis or treatment," that the services "are provided for educational and informational purposes only," and that they do not "substitute for the clinical judgment of a qualified healthcare professional." The API terms of service, last updated March 16, 2024, say the services are "intended and presented only for general educational purposes" and are provided on an "AS-IS" basis.