Although I’m currently a practicing attorney, in a prior career I was a researcher and a published statistical data analyst with the Washington University School of Medicine. I have a master’s degree in public health. And I have grave concerns about the numbers that the State of Illinois (and other states) are using to make decisions about closing down large facets of our economy. The so-called “positivity rate” doesn’t tell us the full story about what is happening in the community with this disease.
As a quick example, if 100 people are tested for COVID-19 right now, it is extremely likely that they are being tested because they are sick, or because they have recently been in close contact with someone who has tested positive. If 6 of those people test positive (a 6% “positivity rate”) or 8 test positive, (an 8% “positivity rate”), does that tell us what the rate of positive tests are in the community? No. Because it is very possible – even probable – that the folks being tested are more likely to have a positive test result. Again, they may have felt sick or have been in close proximity to someone who is. So of course the percent with a positive result can be higher than in the general population. In science, this is called “selection bias,” or “ascertainment bias.” More thorough definitions of selection and ascertainment bias can be found online.
One researcher described exactly this problem in a very recently-published article. According to BMC Med Res Methodol: “A crude measure of population prevalence [of COVID-19] is the fraction of positive tests at any given date. However, this is subject to large ascertainment bias since tests are typically only ordered from symptomatic cases, whereas a large proportion of infected might show little to no symptoms…”
In order to have a valid estimate of the actual prevalence of COVID-19 infection within the community, we would need to look at the test results from a random sample of people. This is more time-consuming, that’s true. But it is possible.
In July, the CDC published the results from the state of Indiana, which tested a random sample of people. The results showed an estimated SARS-CoV-2 prevalence of active or current infection in Indiana of 2.79%. Very importantly, when they looked at folks who had a household member who had previously tested positive for COVID-19, the rate was 33.6%. This means that they estimated about 2.8% of people overall were positive for COVID-19. But, they estimated that about 33.6% of folks were positive, when they had a household member who was also positive. This is why ascertainment bias is so important to understand and to talk about.
It would be unreasonable to suggest that the government should rely on fully-controlled studies or randomly-collected samples in order to make policy decisions during a pandemic. However, relying only on one, skewed measurement like the “positivity rate” is a critical error when making decisions to shut down large portions of our economy. This is especially true when the number of positive folks who actually have symptoms is lower than with many other diseases. Estimates are that as much as 45% of positive cases are asymptomatic, meaning they aren’t sick.
The effects of closing down businesses in our community are devastating. Also, we probably haven’t even begun to understand the non-economic effects of the isolation and job loss resulting from the last few months. For example, one recent study estimated more than a 3-fold increase in depression among Americans during the COVID-19 pandemic. That study also found that lower income, having less than $5,000 in savings, and having exposure to more stressors were associated with greater risk of depression symptoms during COVID-19. Yet we don’t seem to be having a discussion about the effects of closures on the mental health and wellness of the community during these already trying times. Rather, those at the state level seem focused purely on this “positivity rate.”
So, what measurements should we be looking at before we make decisions to close business and force folks out of work? We were originally told that the shutdowns weren’t going to stop the virus, they were just supposed to “flatten the curve” so that the health care system wouldn’t be overwhelmed. The state’s own data show that as of October 3, in the metro east region, 47% of ICU beds and about 76% of ventilators were available. Based upon the concept of flattening the curve, why isn’t our conversation focused upon available healthcare resources, rather than the flawed metric “positivity rate?” At the very least, shouldn’t we be looking at multiple sources of data before we make decisions that are going to effect so many lives in so many ways?
I recently had my first conversation with Dr. Loren Hughes. He is a member of the Madison County Healthcare Advisory Committee and has been practicing medicine locally and working with public health matters for 30 years. He agrees.
According to Dr. Hughes, “[w]hile percent positivity rate helps measure the presence and trend of community spread, the IDPH-recognized 3 to 5% estimated error rate in reporting. The limitations of contract tracing, and selective bias in who is being tested make these numbers very ‘fuzzy.’ The state is using a sharply pointed pen to draw the lines. There are six major metrics being tracked and I would rather see each county in the state evaluated on the total picture rather than centering solely on the positivity rate.”
He continued, “How do we weigh the enormity of social woes that are being seen in frightening numbers. There should be balance.”
It has been years since I was in the research field, but many of these issues are common sense. And, admittedly, this is a complex problem. But it is precisely because this issue is so complex, that it seems to be such a huge mistake to make closure decisions based only on one simple number, the “positivity rate.” We are all in this together, I wish we were all having these discussions and making such serious decisions together as well.
James Craney is the founder of Craney Law Group, with offices in Edwardsville and St. Louis.