Absent friends

In June more than 700,000 American clinicians put questions to a single program, OpenEvidence, an AI tool that answers clinical questions from the medical literature. A study published in JAMA on October 15, built on the company's own logs and co-written by its staff, found that a third of physicians' questions concerned treatment and management, and fewer than one in seven concerned diagnosis. Doctors, in short, are asking software what to do. The authors note that many evaluations of such tools have concentrated on diagnosis instead.

Doctors have made AI part of their daily work, and when its advice is wrong it is their name on the order. Yet the week's other news showed the rules for these tools being drafted in rooms where the companies that sell them, and the insurers that pay for care, have seats and practicing doctors have few. Physicians should claim more of those seats now, and use them to demand evidence for the work the tools actually do.

Consider Medicare. For about a year technology firms have lobbied federal health officials in a Slack chat room run by the Centers for Medicare & Medicaid Services, KFF Health News reported. The workspace has some 1,700 members, of whom only a handful are doctors, patient advocates or hospital representatives. Vendors used it to press for wider access to patients' records, and officials indicated that apps the agency promotes would get priority in a new program paying for chatbot advice and wearable tracking. Dr. Mehmet Oz, who runs the agency, described the spirit of the thing in September: "We wanted to give industry a chance to play in the sandbox of government-supported healthcare."

The rules on what AI may do without a doctor are taking shape in a similar way. The Federation of State Medical Boards, which represents the boards that license physicians, has convened a 22-member workgroup to draft guidance for AI tools that perform clinical functions "with limited or no direct physician supervision." Its members include the American Medical Association and the American Hospital Association, but also Anthropic, an AI developer, and Elevance Health, a large insurer; its discussions are confidential, and STAT reported that the guidance could become the basis for state law. The AMA, to its credit, published a framework of its own this week, holding that "clinical authority should be earned through evidence and maintained through demonstrated performance."

Taken seriously, that principle would catch much of what hospitals already run. In the emergency departments of one New York health system, Epic's alert for pediatric sepsis identified only about a third of the children who met criteria for sepsis within a day of arrival, researchers reported in JAMA Pediatrics. Fewer than one in thirty of its alarms pointed to a child who had sepsis, and among Black children it caught barely more than one in five. Epic says each hospital validates and configures the model for its own patients, and that the study measured it against a different definition of sepsis.

Official approval offers less comfort than it sounds. Federal rules require makers of electronic health records to disclose how they analyze and manage the risks of their predictive AI tools; researchers at Georgetown and MedStar Health found the disclosures of the 35 developers they examined inconsistent and often incomplete, yet every one had been certified as meeting the requirements. Epic's deterioration index, which scores hospitalized adults' risk of decline, won clearance from the Food and Drug Administration in September after years of use. In Epic's own testing, at its medium setting, it caught about half of the white patients who deteriorated but only about a quarter of the Black patients.

Those who pay for care are moving faster still. UnitedHealth Group told investors on October 13 that AI had cut the time its appeals take by about a quarter, and said it will offer AI-powered plan recommendations to Medicare Advantage members during this autumn's enrollment; the figures are the company's own. Patients, meanwhile, take advice from chatbots that no one licenses. Attorneys general in eight states sued OpenAI this week, alleging that ChatGPT misled the public about its safety and harmed children, and researchers at Vanderbilt recently counted 28 patients whose psychosis, by the researchers' reading of behavioral health notes, had been worsened by chatbot use. Where rules are missing, attorneys general and courts write them afterward.

Physicians' own organizations are hardly disinterested, and they already have a seat. The AMA sits on the federation's workgroup, medical societies have a long record of defending their members' turf, and a profession that wants evidence before software may act can be accused of wanting a moat. The charge has some force, but it misreads the present balance. The sellers and payers in these rooms have interests at least as plain, and few of them see how the tools behave at the bedside. The JAMA study shows what no vendor benchmark does, that doctors lean on AI chiefly for treatment decisions, the use that has been least tested, and a demand for published evidence would bind the tools doctors like as tightly as those they fear.

Medical staffs should require, before any AI tool that alerts, triages or advises on treatment is renewed, a local check of how often it is right, and should share the results with their members. State medical societies should press the federation to add practicing physicians from several specialties to its workgroup and to publish its draft for comment before any board adopts it. Medicare should move its conversations with industry into the open, under the rules that govern federal advisory committees, and federal certifiers should reject AI disclosures that disclose little. Insurers that use AI on appeals should report how often its conclusions are overturned. Physicians with views on generative AI should file them with the FDA, whose comment period closes on October 19.

More than 700,000 clinicians a month now put their patients' problems to a piece of software, and physicians' questions are mostly about what to do. Few of the rooms drafting its rules this autumn were built around that fact. The people asking the questions should help write the rules for the answers.