In mid-November, as the United States set records for newly diagnosed COVID-19 cases day after day, the hospital situation in one hard-hit state, Wisconsin, looked concerning but not yet urgent by one crucial measure. The main pandemic data tracking system run by the Department of Health and Human Services (HHS), dubbed HHS Protect, reported that on 16 November, 71% of the state’s hospital beds were filled. Wisconsin officials who rely on the data to support and advise their increasingly strained hospitals might have concluded they had some margin left.
Yet a different federal COVID-19 data system painted a much more dire picture for the same day, reporting 91% of Wisconsin’s hospital beds were filled. That day was no outlier. A Science examination of HHS Protect and confidential federal documents found the HHS data for three important values in Wisconsin hospitals—beds filled, intensive care unit (ICU) beds filled, and inpatients with COVID-19—often diverge dramatically from those collected by the other federal source, from state-supplied data, and from the apparent reality on the ground.
“Our hospitals are struggling,” says Jeffrey Pothof, a physician and chief quality officer for the health system of the University of Wisconsin (UW), Madison. During recent weeks, patients filled the system’s COVID-19 ward and ICU. The university’s main hospital converted other ICUs to treat the pandemic disease and may soon have to turn away patients referred to the hospital for specialized care. Inpatient beds—including those in ICUs—are nearly full across the state. “That’s the reality staring us down,” Pothof says, adding: The HHS Protect numbers “are not real.”
HHS Protect’s problems are a national issue, an internal analysis completed this month by the Centers for Disease Control and Prevention (CDC) shows. That analysis, other federal reports, and emails obtained by Science suggest HHS Protect’s data do not correspond with alternative hospital data sources in many states (see tables, below). “The HHS Protect data are poor quality, inconsistent with state reports, and the analysis is slipshod,” says one CDC source who had read the agency’s analysis and requested anonymity because of fear of retaliation from the Trump administration. “And the pressure on hospitals [from COVID-19] is through the roof.”
Both federal and state officials use HHS Protect’s data to assess the burden of disease across the country and allocate scarce resources, from limited stocks of COVID-19 medicines to personal protective equipment (PPE). Untrustworthy numbers could lead to supply and support problems in the months ahead, as U.S. cases continue to rise during an expected winter surge, according to current and former CDC officials. HHS Protect leaders vigorously defend the system and blame some disparities on inconsistent state and federal definitions of COVID-19 hospitalization. “We have made drastic improvements in the consistency of our data … even from September to now,” says one senior HHS official. (Three officials from the department spoke with Science on the condition that they not be named.)
CDC had a long-running, if imperfect, hospital data tracking system in place when the pandemic started, but the Trump administration and White House Coronavirus Task Force Coordinator Deborah Birx angered many in the agency when they shifted much of the responsibility for COVID-19 hospital data in July to private contractors. TeleTracking Technologies Inc., a small Pittsburgh-based company, now collects most of the data, while Palantir, based in Denver, helps manage the database. At the time, hundreds of public health organizations and experts warned the change could gravely disrupt the government’s ability to understand the pandemic and mount a response.
The feared data chaos now seems a reality, evident when recent HHS Protect figures are compared with public information from states or data documented by another hospital tracking system run by the HHS Office of the Assistant Secretary for Preparedness and Response (ASPR). ASPR manages the Strategic National Stockpile of medicines, PPE—in perilously short supply in many areas—and other pandemic necessities. ASPR collects data nationwide, although it is more limited than what HHS Protect compiles, to help states and hospitals respond to the pandemic.
In Alabama, HHS Protect figures differ by 15% to 30% from daily state COVID-19 inpatient totals. Karen Landers, assistant state health officer, said nearly all of the state’s hospitals report data to HHS via the Alabama Department of Public Health. Although reporting delays sometimes prevent the systems from syncing precisely, Landers says, she cannot account for the sharp differences.
Many state health officials contacted by Science were reluctant to directly criticize HHS Protect or attribute supply or support problems to its data. Landers notes that Alabama relies on its own collected data, rather than HHS Protect’s, for its COVID-19 response. “We are very confident in our data,” she says, because the state reporting system was developed over several years and required little adjustment to add COVID-19. HHS, she adds, has generally been responsive to state requests for medicines and supplies, although Alabama has not always gotten all the PPE it has requested.
Other states, however, say they do rely on HHS Protect. A spokesperson for the Wisconsin Department of Health Services wrote in a response to questions, “When making decisions at the state level we use the HHS Protect data,” but declined to comment about its accuracy. HHS informed Wisconsin officials it distributes scarce supplies based on need indicated by HHS Protect data, the spokesperson wrote.
Pothof says UW’s hospital system has its own sophisticated data dashboard that draws on state, local, and internal sources to plan and cooperate on pandemic response with other hospitals. But small hospitals in Wisconsin—now experiencing shortages of some medicines, PPE, and other supplies—are more dependent on federal support largely based on HHS Protect data. Help might not arrive, Pothof says, if the data show “things look better than they are.”
If the HHS Protect data are suspect, “that’s a very large problem,” says Nancy Cox, former director of CDC’s influenza division and now an affiliated retiree of the agency. If HHS officials use bad data, they will not distribute medicines and supplies equitably, Cox notes, adding: “Undercounting in the hardest hit states means a lower level of care and will result in more severe infections and ultimately in more deaths.”
Birx and the other managers of HHS Protect “really had no idea what they were doing,” says Tom Frieden, CDC director under former President Barack Obama. (Birx declined to comment for this article.) Frieden cautions that ASPR data might also be erroneous—pointing to the need for an authoritative and clear federal source of hospital data. The original CDC system, called the National Healthcare Safety Network (NHSN), should be improved, he said, but it handles nursing home COVID-19 data skillfully and could do the same with hospitals. NHSN is “not just a computer program. It’s a public health program” built over 15 years and based on relationships with individual health facilities, Frieden says. (CDC insiders say HHS officials recently interfered with publication of an analysis showing that NHSN performed well early in the pandemic [see sidebar, below]).
An HHS official says HHS Protect’s data are complex and the department can’t verify any findings in the reports reviewed by Science without conducting its own analysis, which it did not do. But the official says HHS Protect has improved dramatically in the past 2 months and provides consistent and reliable results.
As for the difference between state and HHS Protect data, an HHS official contends state numbers “are always going to be lower” by up to 20%. That’s because hospitals could lose Medicare funding if they do not report to HHS, the official says, but face no penalty for failing to report to the state. So rather than expect identical numbers, HHS looks for state and federal data to reflect the same trajectory—which they do in all cases for COVID-19 inpatient data, according to another confidential CDC analysis of HHS Protect, covering all 50 states.
Yet the same analysis found 27 states recently alternated between showing more or fewer COVID-19 inpatients than HHS Protect—not always just fewer, as HHS says should be the case. Thirty states also showed differences between state and HHS Protect figures that were frequently well above the 20% threshold cited by HHS, and HHS Protect data fluctuated erratically in 21 states (see chart, below).
“Hospital capacity metrics can and should be a national bellwether,” the CDC data expert says. “One important question raised by the discordant data reported by HHS Protect and the states is whether HHS Protect is systematically checking data validity.” HHS has not provided its methodology for HHS Protect data estimates for review by independent experts. But an HHS official says a team of data troubleshooters, including CDC and ASPR field staff, work to resolve anomalies and respond to spikes in cases in a state or hospital.
Out of sync
Tracking hospital inpatients who have COVID-19 has become a crucial measure of the pandemic’s severity. Department of Health and Human Services (HHS) data from the HHS Protect system often diverge sharply from state-supplied data. This chart, drawn from a data analysis from the Centers for Disease Control and Prevention, summarizes some of the similarities and differences for COVID-19 inpatient totals over the past 2 months.
Along with improving trust in its data, HHS Protect needs to make it more accessible, CDC data scientists say. The publicly accessible HHS Protect data are far less complete than the figures in its password-protected database. This effectively hides from public view key pandemic information, such as local supplies of protective equipment.
The site also does not provide graphics highlighting patterns and trends. This might explain, in part, why most media organizations—as well as President-elect Joe Biden’s transition team—instead have relied on state or county websites that vary widely in completeness and quality, or on aggregations such as The Atlantic magazine’s COVID Tracking Project, which collects, organizes, and standardizes state data. (In comparing state and federal data, CDC also used the COVID Tracking Project.)
Frieden and other public health specialists call reliable, clear federal data essential for an effective pandemic response. “The big picture is that we’re coming up to 100,000 hospitalizations within the next few weeks. Hospital systems all over the country are going to be stressed,” Frieden says. “There’s not going to be any cavalry coming over the hill from somewhere else in the country, because most of the country is going to be overwhelmed. We’re heading into a very hard time with not very accurate information systems. And the government basically undermined the existing system.”
HENRY KOTULA 