https://www.medpagetoday.com/infectiousdisease/covid19/88264?xid=fb_o&trw=no&fbclid=IwAR0F6xiRAQ7ngBz4pNozJ2VqWm0-UJqGdlQojfOeyXbPJjbAeYtGL8jbAiw
“The data don’t show anything useful”
Problems with the government’s rationale for authorizing use of convalescent plasma in COVID-19 patients go far beyond the dustup over the purported 35% survival benefit cited by top officials on Sunday, numerous researchers say.
That figure quickly came under fire, leading to an apology from Commissioner Stephen Hahn, MD — but that’s not the only criticism leveled at the FDA’s analysis of the available data.
Much of it came from the Mayo Clinic and FDA expanded access program (EAP), at this point published only as a preprint manuscript. Although a large number of patients were included, the study was observational only, with no untreated control group. That makes the findings merely hypothesis-generating, and can’t offer any firm conclusions.
That’s fine for issuing an emergency use authorization (EUA), but not so much for making claims about survival benefit, independent researchers said.
“It’s not even a question of overstating,” Adam Gaffney, MD, MPH, a critical care doctor and health policy researcher at Harvard Medical School, told MedPage Today. “You can’t state much at all when you don’t have a randomized controlled trial.”
“People have made a big deal of Hahn referring to relative versus absolute risk reduction, but I think that’s less of a big deal,” Gaffney said. “The biggest problem is that the data they are citing … is not randomized. That’s the source of all the problems.”
Hahn took heat for saying that a “35% improvement in survival is a pretty substantial clinical benefit” further explaining that of “100 people who are sick with COVID-19, 35 would have been saved because of the administration of plasma.”
Critics rapidly took to Twitter, stating that the interpretation was incorrect. Hahn was referring to relative risk reduction, not absolute risk reduction. Thus, calculating the number of lives saved — which isn’t something experts recommend doing based on observational data in the first place — would have translated to somewhere more in the ballpark of 5 out of 100.
Moreover, the “risk reduction” came from a comparison of patients treated with high-titer plasma versus those receiving lower-titer preparations. The study offered no basis for concluding how many patients may have been “saved” relative to standard care.
And the 35% reduction in that analysis was for 7-day mortality; the relative reduction at 30 days was only 23%.
Hahn’s recital of the 35% figure “was just PART of the error,” tweeted Vinay Prasad, MD, MPH, of the University of California San Francisco. “The entire comparison is flawed. It is not a suitable control. The data don’t show anything useful.”
“The much broader problem here is the lack of commitment to performing large, national randomized controlled trials,” Gaffney said. “We could have done it for convalescent plasma. Instead, we did the EAP. I understand why people wanted it, but now we don’t know [if convalescent plasma works]. We have a question mark instead of a period.”
Undermining Trust in FDA?
Critics have charged that serious mistakes like Hahn’s misstatement could undermine FDA’s credibility, especially as it faces challenging decisions about potentially approving a vaccine this fall.
“This is playing out in the context of a hyper-politicized moment,” Gaffney said. “It behooves everyone to be extremely cautious in speaking about these things to avoid the appearance of politicization.”
On CBS This Morning on Tuesday, Hahn addressed concerns about politicization by offering reassurance to the “American people that this decision was made based upon sound science and data.”
In response to questions about the timing of the EUA announcement — it came just a day after President Donald Trump tweeted allegations that the “deep state” was holding back access to COVID-19 treatments with Hahn’s Twitter handle cited, and a day before the Republican National Convention got underway — Hahn said the agency had been working on the application for 3 or 4 weeks and was waiting on additional validation data, which were received at the end of last week and over the weekend.
“We’re going to continue to get data and as we’ve done with any other authorization, we will update that decision as new data come,” Hahn said on the news program. His agency initially issued an EUA for hydroxychloroquine, for instance, but later revoked it when the negative randomized trial data became available.
Lack of Access to FDA’s Data Review
Whether the public will ever see the full convalescent plasma data underlying the EUA is another matter. The “Clinical Memorandum” issued as the evidence behind the FDA’s decision glossed over the statistical analysis conducted by the agency; in particular, it made no mention of the 35% relative reduction in deaths.
Another problem with that is the 35% figure’s source isn’t fully clear. Although the EAP preprint manuscript is the most obvious source, Gaffney noted that HHS Secretary Alex Azar said it referred to a subgroup of patients under age 80 who were not on a ventilator. That is not found in the publicly available data. He also pointed to a tweet by FDA spokesperson Emily Miller that contains an agency slide showing a 37% reduction in mortality for non-intubated patients age 80 or under treated within 72 hours who got high-titer convalescent plasma, compared with low-titer product. Neither of those figures is reflected in the EAP manuscript.
The FDA did not return a request by MedPage Today for the full summary of data reviewed by FDA and any independent statistical analysis done by the agency.
Shmuel Shoham, MD, of Johns Hopkins University in Baltimore, said during a press briefing organized by the Infectious Diseases Society of America that “enormous amounts of data have been generated” from the EAP, in which more than 70,000 patients have been treated.
“Some data have been reported in articles and at meetings, but that’s only part of what the FDA — this is their program — has access to,” he said. “The stuff in the public domain is only a fraction of the data they have collected.”
Shoham is on the scientific advisory board of the EAP and is involved in two convalescent plasma clinical trials at Johns Hopkins.
Gaffney said Mayo researchers and FDA reviewers have noted that physicians were blinded to the dose of antibody given in plasma infusions, which he described as a “pseudo-randomization effect. We could use that to make more causal inferences about the effectiveness of antibody titers.”
However, he said there were some significant differences between those who received high-titer versus low-titer antibody, including differences in P-to-F ratio (a measure of inhaled oxygen to blood oxygen) and in those with five or more severe risk factors, suggesting the low-titer group was sicker to begin with than the high-titer group.
Also, patients in the EAP received a variety of other treatments: about half got steroids and 40% were given remdesivir.
“This is why we do randomized controlled trials,” Gaffney said. “Without them it’s very difficult to ensure that the effect you see is the result of the drug, and not the result of patient characteristics.”
Is an Answer Forthcoming?
Several randomized controlled trials of convalescent plasma are underway in the U.S., but the big concern is that wider access to convalescent plasma will limit enrollment. Will clinicians recommend that their patients enroll in a trial in which they might receive placebo? Will patients agree?
For the Hopkins studies, the prevention trial has enrolled 25 people out of a goal of 500, and its outpatient trial has enrolled 50 people of its 600-patient goal.
Liise-anne Pirofski, MD, of Montefiore Medical Center in New York, started a study at the end of April, looking to enroll 300 people. She said the team enrolled the first 150 people quickly, but “then the pandemic began to wane in New York.” With subsequent funding from the NIH, the trial has managed to enroll 190 patients, and has now expanded to four additional sites: New York University, Yale, the University of Miami, and the University of Texas Houston.
Clifton Callaway, MD, PhD, of the University of Pittsburgh Medical Center and lead investigator on the C3PO trial looking at outpatient convalescent plasma, said he hopes the EUA doesn’t discourage participation.
“To the contrary, I believe it should reassure persons considering participation that the FDA feels that convalescent plasma is safe and potentially useful and that the FDA specifically comments: ‘Current data suggest the largest clinical benefit is associated with high-titer units of CCP administered early in the course of disease.’ Giving high-titer convalescent plasma earlier (before you are sick enough to be in the hospital) is exactly what C3PO is testing.”
In addition to determining whether earlier or prophylactic treatment works, Shoham said other unanswered questions include identifying whether other components in plasma are useful therapies and whether low-titer plasma can work at all.
“What everyone agrees on is that the gaps in knowledge that exist can best be addressed by high-quality randomized controlled trials,” he said.
Pirofski said the science and data should be the focus, “rather than the decision and what drove the decision…. I don’t think anyone knows what drove that decision other than the people in that room. Hopefully they know.”
HENRY KOTULA 