(2022-10-24) Epic's Overhaul Of A Flawed Algorithm Shows Why AI Oversight Is A Life Or Death Issue

Epic's overhaul of a flawed algorithm shows why AI oversight is a life-or-death issue. Epic, the nation’s dominant seller of electronic health records (EMR), was bracing for a catastrophe. It was June 2021, and a study about to be published in the Journal of the American Medical Association had found that Epic’s artificial intelligence (AI) tool to predict sepsis, a deadly complication of infection, was prone to missing cases and flooding clinicians with false alarms.

A year later, after a series of investigations by STAT, the company released a re-engineered version of the model it had steadfastly defended. Epic changed the data variables it uses, its definition of sepsis onset, and its guidance for tuning the algorithm to local patients. Even the user guide for the hundreds of hospitals it serves nationwide was entirely different — and twice as long.

Epic is not the only company moving aggressively to sell AI tools to health systems

The Food and Drug Administration (FDA) published a recent guidance that put the regulation of sepsis alerts and other AI predictors squarely within its purview. But it has not made clear whether it will require a review process before those products go on the market, as is required of many algorithms that interpret medical images.

“The lack of standard empiric evidence supporting these algorithms is really bothersome for me,” said Derek Angus, a physician at the University of Pittsburgh Medical Center and expert in treating sepsis

“It’s fundamentally changing the way care is delivered in a hospital.” But Epic’s tool was treated nothing like a drug. It wasn’t subjected to an FDA review process or third-party testing before its initial release

A spokesperson for Epic declined to answer questions about the changes to its sepsis model or the push for stepped-up oversight of AI products.

“If you wait for humans to recognize sepsis, that’s too late,” said Shamim Nemati, a professor of biomedical informatics at UC San Diego School of Medicine

As Epic was preparing to launch its sepsis tool in 2017, several factors were pushing against that kind of slow and methodical evaluation

Hospitals were also under increasing pressure to improve their treatment of sepsis, which kills nearly 270,000 Americans a year, often because it is not discovered in time

At a minimum, most hospitals ran it in the background of their data systems, so they could see how it would respond to individual patients before the alerts were turned on for clinical use. But the extent of the evaluations — whether based on prior data or live patients, and how the effects were analyzed — varied widely.

At UC Health in Colorado, which paired Epic’s sepsis algorithm with its own prediction models, the ratio of false alarms to true positives was about 30 to 1, according to CT Lin, the health system’s chief medical information officer.

“Our doctors’ response to us was, ‘Do you not think we’re running as fast as we can already treating patients who are trying to die in front of us right now?’”

To deal with the high rate of false alarms, UC Health started using a remote team of clinicians to monitor the model’s output and examine patients through a live video feed. When a patient truly seemed to be deteriorating, a remote clinician would call the bedside nurses. But then the bedside nurses, annoyed by the perceived intrusion, began putting coats over the cameras.

In the end, the process produced a positive outcome: Patients received antibiotics to treat sepsis in half the time compared to before the sepsis tool was installed, within an average 40 minutes rather than 80 minutes. The health system estimates that speedier sepsis care saves 211 lives annually.

Liu said throwing out the tools, or abandoning the quest altogether, is not the right move either

The problem, he said, is that so many people have been seduced by the belief that there is such a thing as a perfect global sepsis model that will dramatically improve outcomes — but without special effort to ensure it’s implemented in the right way, at the right moment of treatment.