With the Delta variant now accounting for more than 83 percent of all new COVID cases in the US, daily new case counts more than quadrupling across the month of July, and hospitalizations—particularly in states with low vaccination rates—beginning to climb significantly, we appear to have entered a new and uncertain phase of the pandemic, now being dubbed a “pandemic of the unvaccinated”.
Welcome news, then, that this week the American Hospital Association (AHA) publicly encouraged its members to put in place vaccine mandates for their employees. While several large health systems have taken the lead in implementing vaccine mandates, including Trinity Health, the Livonia, MI-based Catholic system that operates hospitals across 22 states, Phoenix, AZ-based Banner Health, Houston Methodist in Texas, and the academic giant NewYork-Presbyterian, others have been more reticent to compel employees to get vaccinated, citing concerns over employee privacy and the potential for workforce backlash.
The New York Timesreports that a quarter of all hospital employees remain unvaccinated nationwide, with many facilities reporting that more than half of their healthcare workers have not gotten the COVID vaccine. In our discussions with health system executives, one consideration frequently cited is the desire for full Food and Drug Administration (FDA) approval of the new vaccines before mandates are put in place.
In a CNN town hall meeting this week, President Biden suggested that approval could come as soon as the end of August, although other reports point to likely approval much later, potentially not until January of next year. Facing a new variant of the virus that is much more transmissible and possibly more virulent than earlier strains, hospitals—and their patients—can’t afford to wait that long.
For safety’s sake, hospitals should quickly put in place vaccine mandates, with appropriate exceptions.
As the delta variant of the coronavirus spreads, especially among the unvaccinated, the Biden administration is gearing up for a new push to vaccinate the so-called “movable middle”—and some public health experts say FDA could advance that goal by fully approving Covid-19 vaccines.
Analysis reveals toll of US Covid-19 deaths among unvaccinated patients
According to an analysis by the Associated Press, nearly all recent Covid-19 deaths have occurred in unvaccinated individuals.
The AP analysis is based on data from CDC, although CDC has not itself released estimates of the share of Covid-19 deaths among unvaccinated patients.
According to the AP analysis, just 0.8% of Covid-19 deaths in May were among the fully vaccinated. Meanwhile, the share of hospitalized patients who were fully vaccinated was just 0.1% in May, with fewer than 1,200 fully vaccinated people hospitalized out of more than 853,000 hospitalizations.
Meanwhile, according to CDC, 54% of the U.S. population, including 66% of American adults, have received at least one dose of a Covid-19 vaccine, while 46.1% of the total population and 56.8% of American adults have received all required doses.
Two influential advisory groups sent recommendations to Congress calling for a revamp of how health plans are paid in the lucrative Medicare Advantage program, culling how many models CMS tests and curbing high-cost drug approvals.
By many measures, the MA program has been thriving. Enrollment and participation has continued to grow, and in 2021, MA plans’ bids to provide the Medicare benefit declined to a record low: Just 87% of comparable fee-for-service spending in their markets.
But despite that relative efficiency, MA contracting isn’t saving Medicare money — actually, in the 35 years Medicare managed care has been active, it’s never resulted in net savings for the cash-strapped program, James Mathews, executive director of the Medicare Payment Advisory Commission, told reporters in a Tuesday briefing.
MedPAC estimates Medicare actually spends 4% more per capita for beneficiaries in MA plans than those in FFS under the existing benchmark policy.
To save money, Medicare could change how the benchmark, the maximum payment amount for plans, is adjusted for geographic variation, MedPAC said.
Under current policy, Medicare pays MA plans more if they cover an area with lower FFS spending, despite most plans bidding below FFS in these areas. At the same time, plans in areas where FFS spending is higher bid at a lower level relative to their benchmark, and wind up getting higher rebates — the difference between the bid and the benchmark — as a result.
“Because the rebate dollars must be used to provide extra benefits, large rebates result in plans offering a disproportionate level of extra benefits,” MedPAC wrote in its annual report to Congress. “Moreover, as MA rebates increase, a smaller share of those rebates is used for cost-sharing and premium reductions — benefits that have more transparent value and provide an affordable alternative to Medigap coverage.”
The group recommended rebalancing the MA benchmark policy to use a relatively equal blend of per-capita FFS spending in a local area and standardized national FFS spending, which would reduce variation in local benchmarks, and use a rebate of at least 75%. Currently, a plan’s rebate depends on its star rating, and ranges from 65% to 70%.
MedPAC also suggested a discount rate of at least 2% to reduce local and national blended spending amounts.
The group’s simulations suggest the changes would have minimal impact on plan participation or MA enrollees, but could lead to savings in Medicare of about 2 percentage points, relative to current policy.
Finding savings in Medicare, even small ones, is integral for the program’s future, policy experts say. The Congressional Budget Office expects the trust fund that finances Medicare’s hospital benefit will become insolvent by 2024, as — despite perennial warnings from watchdogs and budget hawks — lawmakers have kicked the can on the insurance program’s snowballing deficit for years.
Fewer and more targeted alternative payment models
MedPAC also recommended CMS streamline its portfolio of alternative payment models, implementing a smaller and more targeted suite of the temporary demonstrations designed to work together.
CMS is already undergoing a review of the models, meant to inject more value into healthcare payments, following calls from legislators for more oversight in the program. The agency doesn’t have the most stellar track record: Of the 54 models its Center for Medicare and Medicaid Innovation has trialed since it was launched a decade ago, just four have been permanently encoded in Medicare.
New CMMI head Elizabeth Fowler said earlier this month the agency will likely enact more mandatory models to force the shift toward value, as the ongoing review has resulted in more conscious choices about where it should invest.
In its report, MedPAC pointed out many of CMMI’s models generated gross savings for Medicare, before performance bonuses to providers were shelled out. That suggests the models have the power to change provider practice patterns, but their effects are tricky to measure. Many providers are in multiple models at once, and the same beneficiaries can be shared across models, too.
Additionally, some models set up conflicting incentives. Mathews gave the example of accountable care organizations participating in one model to reduce spending on behalf of an assigned population relative to a benchmark, but its provider participants could also be in certain bundled models with incentives to keep the cost of care per episode low — but not reduce the overall number of episodes themselves.
“The risk of these kinds of inconsistent incentives would be minimized again if the models were developed in a manner where they would work together at the outset,” Mathews said. MedPAC doesn’t have guidance on a specific target number of alternative models, but said it should be a smaller and more strategic number.
Curbing high-cost drugs in Medicaid
Another advisory board, on the Medicaid safety-net insurance program, also released its annual report on Tuesday, recommending Congress mitigate the effect of pricey specialty drugs on state Medicaid programs.
High-cost specialty drugs are increasingly driving Medicaid spending and creating financial pressure on states. The Medicaid and CHIP Payment and Access Commission (MACPAC) didn’t recommend Congress change the requirement that Medicaid cover the drugs, but recommended legislators look into increasing the minimum rebate percentage on drugs approved by the Food and Drug Administration through the accelerated approval pathway, until the clinical benefit of the drugs is verified.
The accelerated approval pathway, which can be used for a drug for a serious or life-threatening illness that provides a therapeutic advantage over existing treatments, allows drugs to come to market more quickly. States have aired concerns about paying high list prices for such drugs when they don’t have a verified clinical benefit.
Several advisors to the FDA have resigned over the decision, as it’s unclear if aducanumab actually has a clinical benefit. What aducanumab does have is an estimated price tag of $56,000 a year, which could place severe stress on taxpayer-funded insurance programs like Medicare and Medicaid if widely prescribed.
MEDPAC also recommended an increase in the additional inflationary rebate on drugs that receive approval from the FDA under the accelerated approval pathway if the manufacturer hasn’t completed the postmarketing confirmatory trial after a specified number of years. Once a drug receives traditional approval, the inflationary rebate would revert back to the standard amounts.
The recommendations would only apply to the price Medicaid pays for the drug and doesn’t change the program’s obligation to cover it.
As we’ve talked to health system executives about the challenges of rolling out COVID vaccines in their communities, one topic keeps coming up: how difficult it’s been to get hospitals’ own workers fully vaccinated. One system told us recently that only 55 percent of their frontline caregivers have opted to get vaccinated, despite early and easy availability, and ongoing encouragement from the hospital’s leaders.
Healthcare workers, it turns out, are just like the general population, bringing the same diversity of perspectives and concerns about vaccination to work with them from their own communities. Vaccine hesitancy is not a new issue for hospital staffers; getting the workforce to take the flu vaccine is an annual struggle for many hospitals.
But given the risks of COVID-19, why not just mandate that hospital employees get the vaccine, as other employers have started to do? We commonly hear two concerns.
One is a labor relations worry: will mandating vaccination cause workers to quit, or make it harder to hire staff in an already difficult market for talent? And given growing concerns about unionization of healthcare workers, will mandatory vaccination become a flashpoint issue?
The second concern is medical liability: can we force workers to get a vaccine that hasn’t been fully approved by the FDA? Would that expose the hospital to legal challenges down the road, if there turn out to be long-term complications from the vaccine?
Our own view is that the first concern is overblown—we suspect vaccine mandates are going to become more and more common as the economy reopens. As to the second, we’re more sympathetic. But once the FDA does grant full approval for the vaccines, we’d hope hospitals will get tougher about vaccine mandates (with the necessary exemptions for health, religious, and other concerns).
At the end of the day, hospitals are in the patient care business, and they should view vaccine mandates—whether for COVID or for influenza—as a patient safety issue, not a workforce engagement issue.
The Food and Drug Administration cleared the first coronavirus vaccine for emergency use in children as young as 12 on Monday, expanding access to the Pfizer-BioNTech shot to adolescents ahead of the next school year and marking another milestone in the nation’s battle with the virus.
The decision that the two-shot regimen is safe and effective for younger adolescents had been highly anticipated by many parents and pediatricians, particularly with the growing gap between what vaccinated and unvaccinated people may do safely. Evidence suggests that schools can function at low risk with prevention measures, such as masks and social distancing.But vaccines are poised to increase confidence in resuming in-person activities and are regarded as pivotal to returning to normalcy.
“Adolescents, especially, have suffered tremendously from the covid pandemic. Even though they’re less likely than adults to be hospitalized or have severe illness, their lives really have been curtailed in many parts of the country,” said Kawsar R. Talaat, an assistant professor of international health at the Johns Hopkins Bloomberg School of Public Health. “A vaccine gives them an extra layer of protection and allows them to go back to being kids.”
Expert advisers to the Centers for Disease Control and Prevention are scheduled to meet Wednesday to recommend how the vaccine should be used in that age group, and the vaccine can be administered as soon as the CDC director signs off on the recommendation.
In a news briefing Monday evening after the announcement, FDA officials said the Pfizer authorization for 12- to 15-year-olds was a straightforward decision because the data showed that the vaccine was safe and that the response to the vaccine was even better than among the 18- to 25-year-olds who got the shots.
Children rarely suffer serious bouts of covid-19, the illness caused by the coronavirus. But there is no way to predict the few who will become dangerously sick or develop a rare, dangerous inflammatory syndrome. Out of more than 581,000 covid-19 deaths in the United States, about 300 have been people under 18 — a tiny fraction of the total. But that exceeds the number of children who die in a bad flu season.
Children appear to be less efficient at spreading the virus, although their role in transmission is still not fully understood — another reason for pediatric vaccinations.
Clinicians also worry that with a new virus with many unknowns, the possibility exists for long-term impacts of infection, even from the mild or asymptomatic courses of illness common among children.
The Pfizer-BioNTech vaccine, already authorized for adolescents 16 and older, was the first to be tested in younger adolescents. The FDA’s decision will provide a potential path for other vaccine-makers to follow, most of which have launched or plan to initiate trials of their vaccines in teenagers and younger children.
The agency based its authorization on a trial of nearly 2,300 adolescents between 12 and 15 years old, half of whom received the same two-shot regimen shown effective and safe in adults. Researchers took blood samples and measured antibody levels triggered by the shots and foundstronger immune responses in the teens than those found in young adults. There were 16 cases of covid-19 in the trial, all of them among adolescents who received a placebo, suggesting the regimen offered similar protection to younger recipients as it does to adults.
Robert W. Frenck Jr., the researcher who led the adolescent trial at Cincinnati Children’s Hospital Medical Center, said the study was designed to test whether it triggered immune responses, not whether it prevented disease. But because of the number of children who became ill in the placebo arm of the trial, it also became evident the vaccine offered robust protection.
“That really points out how much covid there is in the adolescent community,” Frenck said.
The data has not been published or peer-reviewed, but Kathryn M. Edwards, a pediatric infectious-disease specialist at Vanderbilt University Medical Center, said the results announced by Pfizer were “pretty exciting — it looked very effective and the immune responses were really good.”
Edwards said she is comfortable the benefits of vaccinations are clear among teens, noting that while children, in general, are at lower risk of severe covid-19 than adults, older adolescents seem to be more like adults in their risk for covid-19 than the very youngest children.
Audrey Baker, 15, and Sam Baker, 12, rolled up their sleeves for shots in the Pfizer-BioNTech trial at Cincinnati Children’s Hospital Medical Center. Audrey said she had no hesitation about signing up, and misses little things about how life used to be — eating out in restaurants and seeing family.
“I just trusted the science,” Audrey said. “I knew it was tested in adults. I was really just joining, hoping that maybe I could get vaccinated and help out science.”
Sam said he was more hesitant, in part because participating meant many follow-up lab tests. But he decided to do it and thinks he may have gotten the vaccine in the trial because he developed a headache and fever after his second dose.
Their mother, Rachel Baker, said she felt relief because of Sam’s symptoms.
“The biggest benefit has been that I feel a weight off my shoulders,” Rachel said. “We haven’t changed how we do anything. … We’re still masking, we’re still social distancing, but we’re a bit calmer about it all.”
H. Cody Meissner, a pediatrician at Tufts Medical Center and a member of an external advisory committee to the FDA, said he thinks a pediatric vaccine is needed. But he said he would like to see more safety data because the messenger RNA technology at the core of vaccines from Pfizer-BioNTech and the biotechnology company Moderna does not have a long, established safety record, and its first large-scale use began in December.
Meissner abstained from the December vote that overwhelmingly recommended authorization of the Pfizer-BioNTech vaccine for people 16 and older, because he thought the vaccine should be authorized in people 18 and older.
“For those who are eager to get it, it’s important for them to understand that this is very rarely a severe disease in young adolescents, number one, and this is an entirely new vaccine,” Meissner said. “I just don’t want people to get too swept up in fear of hospitalization and death from covid-19 for the first few decades of life.”
But many other physicians take comfort knowing that 250 million shots of messenger RNA vaccine have been given in the United States alone.Serious side effects, such as a risk of anaphylaxis, are extremely rare. Because the trial in teens was an “immune bridging” trial designed to test whether the vaccine triggered immune responses similar to those in adults, researchers did not need to recruit tens of thousands of people to see if those who received a vaccine were protected against illness. The immunebridging technique is commonly used to expand access to vaccines that have been proved effective and safe to adolescents or other populations.
The expansion of eligibility to children will probably ignite debates in families about when to get vaccinated, and among policymakers about whether it should be required.
Dorit Reiss, a law professor focused on vaccine policy at the University of California Hastings College of Law, said she thinks it is unlikely children will be mandated to receive a coronavirus shot until the vaccines win full approval and not just emergency use authorization.
She predicted that acceptance of the vaccine will evolve as more children are vaccinated and depend on the state of the pandemic. She noted that when vaccines are introduced, the rollout often starts slowly before accelerating.
“Nervousness about a new vaccine is normal, especially when it’s for kids,” Reiss said. “Parents that are nervous now might feel different in a few months, once their friends’ kids have gotten vaccinated. And the views of the kids are also going to matter — if teens are going to think this is going to make their lives easier.”
Opening up vaccinations to children may sharpen a debate unfolding globally about the equity of vaccine access. Talaat said that while she can’t wait for her kids to have access to a vaccine, she is troubled by the global inequities as high-risk front-line workers or older people still don’t have access to vaccines in countries where the coronavirus is out of control.
Moderna announced Thursday that an initial analysis of its teen trial found its vaccine was 96 percent effective among participants who received at least one dose. Moderna is in discussions with regulators about the data. Pfizer-BioNTech and Moderna are testing their vaccines in children as young as infants. Johnson & Johnson is planning pediatric trials of its single-shot vaccine.
Trials in younger children are expected to take longer, because researchers must step down gradually in age and determine a safe and effective dose. William Gruber, senior vice president of vaccine clinical research and development at Pfizer, said data from tests in children as young as 2 years old may be available by September or October, with data on children as young as 6 months possible by the end of the year.
Within each age category, a separate risk-benefit assessment may take place. In the youngest children, given the low risk from the coronavirus, side effects may figure more prominently into the analysis, for example. Researchers may end up choosing a lower dose of vaccine. The understanding of children’s role in transmission may also evolve and help guide vaccine use and public policy.
“We are proceeding carefully, cautiously,” Edwards said. “We’re using the same rigid guidelines we use in all vaccines, and we take this very seriously. I think as time goes on and more information becomes available, some of the questions may be easier to address.”
One hesitates to elevate obviously bad arguments, even to point out how bad they are. This is a conundrum that comes up a lot these days, as members of the media measure the utility of reporting on bad faith, disingenuous or simply bizarre claims.
If someone were to insist, for example, that they were not going to get the coronavirus vaccine solely to spite the political left, should that claim be elevated? Can we simply point out how deranged it is to refuse a vaccine that will almost certainly end an international pandemic simply because people with whom you disagree think that maybe this is a good route to end that pandemic? If someone were to write such a thing at some attention-thirsty website, we certainly wouldn’t want to link to it, leaving our own readers having to figure out where it might be found should they choose to do so.
In this case, it’s worth elevating this argument (which, to be clear, is actually floating out there) to point out one of the myriad ways in which the effort to vaccinate as many adults as possible has become interlaced with partisan politics. As the weeks pass and demand for the vaccine has tapered off, the gap between Democratic and Republican interest in being vaccinated seems to be widening — meaning that the end to the pandemic is likely to move that much further into the future.
Consider, for example, the rate of completed vaccinations by county, according to data compiled by CovidActNow. You can see a slight correlation between how a county voted in 2020 — the horizontal axis — and the density of completed vaccinations, shown on the vertical. There’s a greater density of completed vaccinations on the left side of the graph than on the right.
If we shift to the percentage of the population that’s received even one dose of the vaccine, the effect is much more obvious.
This is a relatively recent development. At the beginning of the month, the density of the population that had received only one dose resulted in a graph that looked much like the current density of completed doses.
If we animate those two graphs, the effect is obvious. In the past few weeks, the density of first doses has increased much faster in more-Democratic counties.
If we group the results of the 2020 presidential contest into 20-point buckets, the pattern is again obvious.
It’s not a new observation that Republicans are less willing to get the vaccine; we’ve reported on it repeatedly. What’s relatively new is how that hesitance is showing up in the actual vaccination data.
A Post-ABC News poll released on Monday showed that this response to the vaccine holds even when considering age groups. We’ve known for a while that older Americans, who are more at risk from the virus, have been more likely to seek the vaccine. But even among seniors, Republicans are significantly more hesitant to receive the vaccine than are Democrats.
This is a particularly dangerous example of partisanship. People 65 or older have made up 14 percent of coronavirus infections, according to federal data, but 81 percent of deaths. That’s among those for whom ages are known, a subset (though a large majority) of overall cases. While about 1.8 percent of that overall group has died, the figure for those aged 65 and over is above 10 percent.
As vaccines have been rolled out across the country, you can see how more-heavily-blue counties have a higher density of vaccinations in many states.
This is not a universal truth, of course. Some heavily Republican counties have above-average vaccination rates. (About 40 percent of counties that preferred former president Donald Trump last year are above the average in the CovidActNow data. The rate among Democratic counties is closer to 80 percent.) But it is the case that there is a correlation between how a county voted and how many of its residents have been vaccinated. It is also the case that the gap between red and blue counties is widening.
Given all of that, it probably makes sense to point out that an argument against vaccines based on nothing more than “lol libs will hate this” is an embarrassing argument to make.
The CDC and FDA on Friday lifted the recommended pause on use of Johnson & Johnson’s coronavirus vaccine, saying the benefits of the shot outweigh the risk of a rare blood clot disorder.
Why it matters: The move clears the way for states to immediately resume administering the one-shot vaccine.
The Johnson & Johnson shot had been seen as an important tool to fill gaps in the U.S. vaccination effort. But between the pause in its use and repeated manufacturing problems, its role in that effort is shrinking.
Driving the news:J&J shots have been paused for about two weeks, in response to reports that they may have caused serious blood clots in a small number of patients.
Only six people had experienced those blood clots at the time of the pause. The CDC said Friday that there have been nine additional cases.
Regulators said the number is small enough to safely resume the use of J&J’s vaccine.
What they’re saying: “Safety is our top priority. This pause was an example of our extensive safety monitoring working as they were designed to work — identifying even these small number of cases,” said acting FDA Commissioner Janet Woodcock.
“We’ve lifted the pause based on the FDA and CDC’s review of all available data and in consultation with medical experts and based on recommendations from the CDC’s Advisory Committee on Immunization Practices,” she said.
“We are confident that this vaccine continues to meet our standards for safety, effectiveness and quality.”
What’s next: Regulators said health care providers administering the shot and vaccine recipients should review revised fact sheets about the J&J vaccine, which includes information about the rare blood clot disorder.
That heightened attention is important because the standard treatment for blood clots can make this particular type of clot worse.
Yes, but:J&J was already a relatively small part of the overall domestic vaccination effort, in part because the company missed some of its early manufacturing targets.
Multiple problems have since emerged at a Baltimore facility that makes a key ingredient for the vaccine, which could sideline production for weeks.
Of 26 health systems surveyed by MedCity News, nearly half used automated tools to respond to the Covid-19 pandemic, but none of them were regulated. Even as some hospitals continued using these algorithms, experts cautioned against their use in high-stakes decisions.
A year ago, Michigan Medicine faced a dire situation. In March of 2020, the health system predicted it would have three times as many patients as its 1,000-bed capacity — and that was the best-case scenario. Hospital leadership prepared for this grim prediction by opening a field hospital in a nearby indoor track facility, where patients could go if they were stable, but still needed hospital care. But they faced another predicament: How would they decide who to send there?
Two weeks before the field hospital was set to open, Michigan Medicine decided to use a risk model developed by Epic Systems to flag patients at risk of deterioration. Patients were given a score of 0 to 100, intended to help care teams determine if they might need an ICU bed in the near future. Although the model wasn’t developed specifically for Covid-19 patients, it was the best option available at the time, said Dr. Karandeep Singh, an assistant professor of learning health sciences at the University of Michigan and chair of Michigan Medicine’s clinical intelligence committee. But there was no peer-reviewed research to show how well it actually worked.
Researchers tested it on over 300 Covid-19 patients between March and May. They were looking for scores that would indicate when patients would need to go to the ICU, and if there was a point where patients almost certainly wouldn’t need intensive care.
“We did find a threshold where if you remained below that threshold, 90% of patients wouldn’t need to go to the ICU,” Singh said. “Is that enough to make a decision on? We didn’t think so.”
But if the number of patients were to far exceed the health system’s capacity, it would be helpful to have some way to assist with those decisions.
“It was something that we definitely thought about implementing if that day were to come,” he said in a February interview.
Thankfully, that day never came.
The survey Michigan Medicine is one of 80 hospitals contacted by MedCity News between January and April in a survey of decision-support systems implemented during the pandemic. Of the 26 respondents, 12 used machine learning tools or automated decision systems as part of their pandemic response. Larger hospitals and academic medical centers used them more frequently.
Faced with scarcities in testing, masks, hospital beds and vaccines, several of the hospitals turned to models as they prepared for difficult decisions. The deterioration index created by Epic was one of the most widely implemented — more than 100 hospitals are currently using it — but in many cases, hospitals also formulated their own algorithms.
They built models to predict which patients were most likely to test positive when shortages of swabs and reagents backlogged tests early in the pandemic. Others developed risk-scoring tools to help determine who should be contacted first for monoclonal antibody treatment, or which Covid patients should be enrolled in at-home monitoring programs.
MedCity News also interviewed hospitals on their processes for evaluating software tools to ensure they are accurate and unbiased. Currently, the FDA does not require some clinical decision-support systems to be cleared as medical devices, leaving the developers of these tools and the hospitals that implement them responsible for vetting them.
Among the hospitals that published efficacy data, some of the models were only evaluated through retrospective studies. This can pose a challenge in figuring out how clinicians actually use them in practice, and how well they work in real time. And while some of the hospitals tested whether the models were accurate across different groups of patients — such as people of a certain race, gender or location — this practice wasn’t universal.
As more companies spin up these models, researchers cautioned that they need to be designed and implemented carefully, to ensure they don’t yield biased results.
An ongoing review of more than 200 Covid-19 risk-prediction models found that the majority had a high risk of bias, meaning the data they were trained on might not represent the real world.
“It’s that very careful and non-trivial process of defining exactly what we want the algorithm to be doing,” said Ziad Obermeyer, an associate professor of health policy and management at UC Berkeley who studies machine learning in healthcare. “I think an optimistic view is that the pandemic functions as a wakeup call for us to be a lot more careful in all of the ways we’ve talked about with how we build algorithms, how we evaluate them, and what we want them to do.”
Algorithms can’t be a proxy for tough decisions Concerns about bias are not new to healthcare. In a paper published two years ago, Obermeyer found a tool used by several hospitals to prioritize high-risk patients for additional care resources was biased against Black patients. By equating patients’ health needs with the cost of care, the developers built an algorithm that yielded discriminatory results.
More recently, a rule-based system developed by Stanford Medicine to determine who would get the Covid-19 vaccine first ended up prioritizing administrators and doctors who were seeing patients remotely, leaving out most of its 1,300 residents who had been working on the front lines. After an uproar, the university attributed the errors to a “complex algorithm,” though there was no machine learning involved.
Both examples highlight the importance of thinking through what exactly a model is designed to do — and not using them as a proxy to avoid the hard questions.
“The Stanford thing was another example of, we wanted the algorithm to do A, but we told it to do B. I think many health systems are doing something similar,” Obermeyer said. “You want to give the vaccine first to people who need it the most — how do we measure that?”
The urgency that the pandemic created was a complicating factor. With little information and few proven systems to work with in the beginning, health systems began throwing ideas at the wall to see what works. One expert questioned whether people might be abdicating some responsibility to these tools.
“Hard decisions are being made at hospitals all the time, especially in this space, but I’m worried about algorithms being the idea of where the responsibility gets shifted,” said Varoon Mathur, a technology fellow at NYU’s AI Now Institute, in a Zoom interview. “Tough decisions are going to be made, I don’t think there are any doubts about that. But what are those tough decisions? We don’t actually name what constraints we’re hitting up against.”
The wild, wild west There currently is no gold standard for how hospitals should implement machine learning tools, and little regulatory oversight for models designed to support physicians’ decisions, resulting in an environment that Mathur described as the “wild, wild west.”
How these systems were used varied significantly from hospital to hospital.
Early in the pandemic, Cleveland Clinic used a model to predict which patients were most likely to test positive for the virus as tests were limited. Researchers developed it using health record data from more than 11,000 patients in Ohio and Florida, including 818 who tested positive for Covid-19. Later, they created a similar risk calculator to determine which patients were most likely to be hospitalized for Covid-19, which was used to prioritize which patients would be contacted daily as part of an at-home monitoring program.
Initially, anyone who tested positive for Covid-19 could enroll in this program, but as cases began to tick up, “you could see how quickly the nurses and care managers who were running this program were overwhelmed,” said Dr. Lara Jehi, Chief Research Information Officer at Cleveland Clinic. “When you had thousands of patients who tested positive, how could you contact all of them?”
While the tool included dozens of factors, such as a patient’s age, sex, BMI, zip code, and whether they smoked or got their flu shot, it’s also worth noting that demographic information significantly changed the results. For example, a patient’s race “far outweighs” any medical comorbidity when used by the tool to estimate hospitalization risk, according to a paper published in Plos One. Cleveland Clinic recently made the model available to other health systems.
Others, like Stanford Health Care and 731-bed Santa Clara County Medical Center, started using Epic’s clinical deterioration index before developing their own Covid-specific risk models. At one point, Stanford developed its own risk-scoring tool, which was built using past data from other patients who had similar respiratory diseases, such as the flu, pneumonia, or acute respiratory distress syndrome. It was designed to predict which patients would need ventilation within two days, and someone’s risk of dying from the disease at the time of admission.
Stanford tested the model to see how it worked on retrospective data from 159 patients that were hospitalized with Covid-19, and cross-validated it with Salt Lake City-based Intermountain Healthcare, a process that took several months. Although this gave some additional assurance — Salt Lake City and Palo Alto have very different populations, smoking rates and demographics — it still wasn’t representative of some patient groups across the U.S.
“Ideally, what we would want to do is run the model specifically on different populations, like on African Americans or Hispanics and see how it performs to ensure it’s performing the same for different groups,” Tina Hernandez-Boussard, an associate professor of medicine, biomedical data science and surgery at Stanford, said in a February interview. “That’s something we’re actively seeking. Our numbers are still a little low to do that right now.”
Stanford planned to implement the model earlier this year, but ultimately tabled it as Covid-19 cases fell.
‘The target is moving so rapidly’ Although large medical centers were more likely to have implemented automated systems, there were a few notable holdouts. For example, UC San Francisco Health, Duke Health and Dignity Health all said they opted not to use risk-prediction models or other machine learning tools in their pandemic responses.
“It’s pretty wild out there and I’ll be honest with you — the dynamics are changing so rapidly,” said Dr. Erich Huang, chief officer for data quality at Duke Health and director of Duke Forge. “You might have a model that makes sense for the conditions of last month but do they make sense for the conditions of next month?”
That’s especially true as new variants spread across the U.S., and more adults are vaccinated, changing the nature and pace of the disease. But other, less obvious factors might also affect the data. For instance, Huang pointed to big differences in social mobility across the state of North Carolina, and whether people complied with local restrictions. Differing social and demographic factors across communities, such as where people work and whether they have health insurance, can also affect how a model performs.
“There are so many different axes of variability, I’d feel hard pressed to be comfortable using machine learning or AI at this point in time,” he said. “We need to be careful and understand the stakes of what we’re doing, especially in healthcare.”
Leadership at one of the largest public hospitals in the U.S., 600-bed LAC+USC Medical Center in Los Angeles, also steered away from using predictive models, even as it faced an alarming surge in cases over the winter months.
At most, the hospital used alerts to remind physicians to wear protective equipment when a patient has tested positive for Covid-19.
“My impression is that the industry is not anywhere near ready to deploy fully automated stuff just because of the risks involved,” said Dr. Phillip Gruber, LAC+USC’s chief medical information officer. “Our institution and a lot of institutions in our region are still focused on core competencies. We have to be good stewards of taxpayer dollars.”
When the data itself is biased Developers have to contend with the fact that any model developed in healthcare will be biased, because the data itself is biased; how people access and interact with health systems in the U.S. is fundamentally unequal.
How that information is recorded in electronic health record systems (EHR) can also be a source of bias, NYU’s Mathur said. People don’t always self-report their race or ethnicity in a way that fits neatly within the parameters of an EHR. Not everyone trusts health systems, and many people struggle to even access care in the first place.
“Demographic variables are not going to be sharply nuanced. Even if they are… in my opinion, they’re not clean enough or good enough to be nuanced into a model,” Mathur said.
The information hospitals have had to work with during the pandemic is particularly messy. Differences in testing access and missing demographic data also affect how resources are distributed and other responses to the pandemic.
“It’s very striking because everything we know about the pandemic is viewed through the lens of number of cases or number of deaths,” UC Berkeley’s Obermeyer said. “But all of that depends on access to testing.”
At the hospital level, internal data wouldn’t be enough to truly follow whether an algorithm to predict adverse events from Covid-19 was actually working. Developers would have to look at social security data on mortality, or whether the patient went to another hospital, to track down what happened.
“What about the people a physician sends home — if they die and don’t come back?” he said.
Researchers at Mount Sinai Health System tested a machine learning tool to predict critical events in Covid-19 patients — such as dialysis, intubation or ICU admission — to ensure it worked across different patient demographics. But they still ran into their own limitations, even though the New York-based hospital system serves a diverse group of patients.
They tested how the model performed across Mount Sinai’s different hospitals. In some cases, when the model wasn’t very robust, it yielded different results, said Benjamin Glicksberg, an assistant professor of genetics and genomic sciences at Mount Sinai and a member of its Hasso Plattner Institute for Digital Health.
They also tested how it worked in different subgroups of patients to ensure it didn’t perform disproportionately better for patients from one demographic.
“If there’s a bias in the data going in, there’s almost certainly going to be a bias in the data coming out of it,” he said in a Zoom interview. “Unfortunately, I think it’s going to be a matter of having more information that can approximate these external factors that may drive these discrepancies. A lot of that is social determinants of health, which are not captured well in the EHR. That’s going to be critical for how we assess model fairness.”
Even after checking for whether a model yields fair and accurate results, the work isn’t done yet. Hospitals must continue to validate continuously to ensure they’re still working as intended — especially in a situation as fast-moving as a pandemic.
A bigger role for regulators All of this is stirring up a broader discussion about how much of a role regulators should have in how decision-support systems are implemented.
Of the hospitals surveyed by MedCity News, none of the models they developed had been cleared by the FDA, and most of the external tools they implemented also hadn’t gone through any regulatory review.
“My experience suggests that most models are put into practice with very little evidence of their effects on outcomes because they are presumed to work, or at least to be more efficient than other decision-making processes,” Kellie Owens, a researcher for Data & Society, a nonprofit that studies the social implications of technology, wrote in an email. “I think we still need to develop better ways to conduct algorithmic risk assessments in medicine. I’d like to see the FDA take a much larger role in regulating AI and machine learning models before their implementation.”
Developers should also ask themselves if the communities they’re serving have a say in how the system is built, or whether it is needed in the first place. The majority of hospitals surveyed did not share with patients if a model was used in their care or involve patients in the development process.
In some cases, the best option might be the simplest one: don’t build.
In the meantime, hospitals are left to sift through existing published data, preprints and vendor promises to decide on the best option. To date, Michigan Medicine’s paper is still the only one that has been published on Epic’s Deterioration Index.
Care teams there used Epic’s score as a support tool for its rapid response teams to check in on patients. But the health system was also looking at other options.
“The short game was that we had to go with the score we had,” Singh said. “The longer game was, Epic’s deterioration index is proprietary. That raises questions about what is in it.”