The three policy changes that may end the pandemic — and avoid the next pandemic
Changing the case definition, the cycle thresholds for PCR testing, and not testing asymptomatic people may end the never-ending pandemic, and will dramatically reduce the chance of a new pandemic
As a policy attorney for 20 years, I’ve come to understand how important policy choices can be. Definitions of key terms, what data to include and what to exclude, how to ensure data quality, how to test people and when not to test people, are all highly important aspects of public health policy.
In tracking the Covid-19 pandemic policy choices over the last year and a half, and in working with a team of medical professionals with deep experience in testing and health policy, I’ve come to the following lessons learned, applicable to the ongoing coronavirus pandemic and for the next pandemic.
If these three policy choices are heeded they will go far in ending (finally) this pandemic and also in avoiding the next pandemic.
1. Require consideration of symptoms as well as positive lab tests for a “confirmed case”
For almost the first time in history, in April of 2020, CDC defined a “confirmed case” of Covid-19 as a positive lab test result only. No consideration of symptoms is required.
This was a first for a respiratory ailment (see also Cohen et al. 2020 noting the same). Previously, including for SARS and many other similar outbreaks, the definition of a confirmed case included consideration of symptoms as well as lab tests.
The CDC provides a page on the history of the surveillance case definitions for various diseases. Uniform case definitions for a number of diseases were first established by CDC and CSTE in 1990. The last time the general criteria were modified was in 1997. The 1997 report states:
“These case definitions are to be used for identifying and classifying cases, both of which are often done retrospectively, for national reporting purposes. They should not be used as criteria for public health action.”
This admonition was not heeded with respect to the Covid-19 pandemic. But it should be for the next one.
Recent case definitions from CDC on, for example, the SARS outbreak in 2003 and the H1N1 in 2008, required clinical symptoms plus laboratory confirmation for a case to be “confirmed”. The CDC’s 2003 case definition for SARS requires (p. 2): “Clinically compatible illness (i.e., early, mild-to-moderate, or severe) that is laboratory confirmed.”
The influenza (flu) case definition, last updated in 2012, also requires both clinical and lab evidence for a confirmed case: “A case that meets the clinical and laboratory evidence criteria.”
As mentioned above, the CDC’s “confirmed case” definition for Covid-19 requires only “confirmatory laboratory evidence.”
So the 2020 case definition for Covid-19 was a substantial break from the policies in place for decades prior to 2020. This change in case definition alone played a major role in how the pandemic unfolded, as we’ll see below.
In sum, the new CDC Covid-19 surveillance case definition adopted a very loose set of criteria when it was adopted in April of 2020, setting up the U.S. for a self-reinforcing chain of pandemic data that rests largely on this new case definition and a number of laboratory tests that are in many cases highly inaccurate.
Any future respiratory ailment should require consideration of symptoms alongside lab tests for a “confirmed case.”
2. Set cycle threshold at a reasonable level for PCR testing
NPR reported in May of 2021 about their obtaining an internal CDC review finding major problems with the development of the CDC’s PCR Covid-19 test in early 2020:
The review bears out reporting from NPR in November that revealed a small infectious disease lab at the CDC was aware the coronavirus test kit it had developed could fail 33% of the time but released it anyway. Under normal circumstances, that kind of failure rate would stop a test in its tracks.
A 33% failure rate, as we’ll see below, is far worse than it sounds. But it’s not just test design issues that are a problem. It’s also how the test is being used.
The New York Times dropped a quiet bombshell at the end of August of 2020 with a story titled “Your coronavirus test is positive. Maybe it shouldn’t be.” It quotes a number of academics and researchers who express strong concerns about how the “gold standard” PCR tests for the coronavirus are being applied.
In short, the tests are being applied in a way that amplifies their sensitivity far beyond what is warranted for tracking active infections of Covid-19 — which is the purpose of the PCR test.
Here’s the key quote from the article:
In three sets of testing data that include cycle thresholds, compiled by officials in Massachusetts, New York and Nevada, up to 90 percent of people testing positive carried barely any virus, a review by The Times found.
On Thursday, the United States recorded 45,604 new coronavirus cases, according to a database maintained by The Times. If the rates of contagiousness in Massachusetts and New York were to apply nationwide, then perhaps only 4,500 of those people may actually need to isolate and submit to contact tracing.
The article never uses the term “false positive” but the primary researchers who track this issue do use this term. Jefferson et al. 2020 “Viral cultures for COVID-19 infectivity assessment — a systematic review (Update 3)”, the third update issued by Jefferson’s team calibrating the accuracy of PCR tests (by using viral cultures) through comprehensive tracking of published test results around the world, concludes (emphasis added):
Prospective routine testing of reference and culture specimens are necessary for each country involved in the pandemic to establish the usefulness and reliability of PCR for Covid-19 and its relation to patients’ factors. Infectivity is related to the date of onset of symptoms and cycle threshold level. A binary Yes/No approach to the interpretation RT-PCR unvalidated against viral culture will result in false positives with segregation of large numbers of people who are no longer infectious and hence not a threat to public health.
In layman’s terms, the PCR tests are being used overly aggressively to amplify a very small signal, which is probably in most cases dead viral fragments, through an excessive number of cycles, or simply background human or microbial genetic material (due to insufficiently specific PCR test probes and primers).
By going beyond the now-established cycle threshold (CT), for detecting live infections, of 25–30 cycles, the PCR tests are creating an artificial positive test result through excessive amplification.
Another quote from the NY Times article:
“I’m really shocked that it could be that high — the proportion of people with high C.T. value results,” said Dr. Ashish Jha, director of the Harvard Global Health Institute. “Boy, does it really change the way we need to be thinking about testing.”
The article adds, quoting another virologist:
Any test with a cycle threshold above 35 is too sensitive, agreed Juliet Morrison, a virologist at the University of California, Riverside. “I’m shocked that people would think that 40 could represent a positive,” she said.
A more reasonable cutoff would be 30 to 35, she added. Dr. Mina said he would set the figure at 30, or even less. Those changes would mean the amount of genetic material in a patient’s sample would have to be 100-fold to 1,000-fold that of the current standard for the test to return a positive result — at least, one worth acting on.
Dr. Mina is Harvard Medical School epidemiologist Michael Mina, an assistant professor at the Center for Communicable Diseases. He also told the NY Times:
In Massachusetts, from 85 to 90 percent of people who tested positive in July with a cycle threshold of 40 would have been deemed negative if the threshold were 30 cycles, Dr. Mina said. “I would say that none of those people should be contact-traced, not one,” he said.
If 85–90% of positive tests “would have been deemed negative,” these are false positive test results. Strong support for this position may be found in CDC’s May 2021 guidance for “breakthrough cases” tracking and reporting. A breakthrough case is a Covid-19 case in someone who has been fully vaccinated. It’s important to track these cases in order to know how effective the vaccines are.
CDC’s May 2021 guidance stated that CDC was changing its tracking approach to include only hospitalizations and deaths, and would no longer publicly track cases if they didn’t lead to hospitalization or death.
Moreover, CDC’s guidance stated that any positive test sample from a breakthrough case that is sent to CDC for genetic sequencing (to test for variants) should not be sent if the cycle threshold was higher than 28. The website states: “For cases with a known RT-PCR cycle threshold (Ct) value, submit only specimens with Ct value ≤28 to CDC for sequencing.”
CDC’s guidance for sending samples for sequencing in the context of normal cases does not contain this same CT threshold guidance. In fact, there is no mention of cycle threshold in this guidance.
Accordingly, CDC decided that in the context of breakthrough cases a cycle threshold of 28 is sufficiently high to ensure a true positive.
As we’ve just seen, reducing the cycle threshold from 40 to 30 would reduce positive results by 85–90%. If we reduce that to a more reasonable 28 (based on CDC’s recent guidance for breakthrough cases) it would reduce the positive PCR test results by over 90%.
The 2003 SARS outbreak also led to virus-specific PCR tests being developed. But the guidance (Jiang et al. 2004) for those tests was for 25 cycle thresholds — a significant reason that that pandemic was quickly brought under control (because there weren’t 90% or more false positives leading to panic and poor downstream policy choices).
A reasonable case definition, that requires consideration of symptoms as well as a positive lab test result, plus a reasonable cycle threshold for PCR testing, will do much to avoid an outbreak of false positives in the next pandemic.
3. Do not test asymptomatic people
Last, but not least, asymptomatic people should rarely or never be tested.
It’s well-known that widespread testing of people with a low probability of having the disease at issue will lead to high levels of false positives. This is known as the “false positive paradox.” It’s a paradox because even apparently quite accurate tests can lead to high levels of false positives when used widely in a population with low actual prevalence.
CDC’s 2004 guidance from the SARS pandemic, for example, stated: “To decrease the possibility of a false-positive result, testing should be limited to patients with a high index of suspicion for having SARS-CoV disease.”
WHO and CDC did, however, recommend widespread testing of asymptomatic people early in the Covid-19 pandemic, but revised this guidance in August of 2020, only to reverse course again after public and expert pushback — clearly, there was a lot of internal debate about this important issue in these agencies.
CDC’s most recent (March 2021) guidance does again recommend screening testing of asymptomatics, despite the widely known issues regarding this policy. This guidance states: “Rapid, point-of care serial screening can identify asymptomatic cases and help interrupt SARS-CoV-2 transmission. This is especially important when community risk or transmission levels are substantial or high.”
The prudent policy in this situation is, however, to not test asymptomatics because such testing leads to extremely high levels of false positives even with highly accurate tests. As we’ve seen above, the available PCR and antigen tests are not close to being highly accurate. Inaccurate tests combined with widespread testing of asymptomatics leads to catastrophically high levels of false positives.
Here’s why testing asymptomatics is such a bad idea: even a test with a very high 99% accuracy rate used to screen asymptomatic populations, with a low background rate of actual infection, will yield high levels of false positives. And the background rate of actual infection, even during “spikes,” has always been relatively low. For example, Baden et al., 2020, found a 0.6% background positive test result in the 30,420 clinical trial participants for the Moderna virus, so assuming 1%, as I do in the results in Figure 1, is a high figure.
An essay in The Guardian newspaper explains this issue well, and it’s based on Bayesian probability, which sounds complex but is actually pretty simple:
Imagine you undergo a test for a rare disease. The test is amazingly accurate: if you have the disease, it will correctly say so 99% of the time; if you don’t have the disease, it will correctly say so 99% of the time.
But the disease in question is very rare; just one person in every 10,000 has it. This is known as your “prior probability”: the background rate in the population.
So now imagine you test 1 million people. There are 100 people who have the disease: your test correctly identifies 99 of them. And there are 999,900 people who don’t: your test correctly identifies 989,901 of them.
But that means that your test, despite giving the right answer in 99% of cases, has told 9,999 people that they have the disease, when in fact they don’t. So if you get a positive result, in this case, your chance of actually having the disease is 99 in 10,098, or just under 1%. If you took this test entirely at face value, then you’d be scaring a lot of people, and sending them for intrusive, potentially dangerous medical procedures, on the back of a misdiagnosis.
Without knowing the prior probability, you don’t know how likely it is that a result is false or true. If the disease was not so rare — if, say, 1% of people had it — your results would be totally different. Then you’d have 9,900 false positives, but also 9,990 true positives. So if you had a positive result, it would be more than 50% likely to be true.
Figure 1 is based on the British Medical Journal (BMJ)’s Covid-19 test accuracy interactive calculator (go ahead, play with it yourself; it’s fun! And very educational).
In populating the three cells in the calculator (at the top of the image) I’ve assumed a 1% background prevalence of active infection, which is as mentioned above an extremely high level of active infection (testing, such as antibody testing, of populations for past infections is a different kind of testing and is designed to pick up people that have had the infection at any point in the past, not in a particular snapshot in time, as is the case for PCR or antigen tests). I’ve also assumed 58% sensitivity and 99% specificity, which are the findings of a recent metastudy combining 64 published reviews of antigen test accuracy.
We get fully 50% false positives in this scenario (1/2 positives are false positives) — even with a 99% specificity test. And zero false negatives.
50% is the same as chance. In other words, this 99% specificity test can in this scenario do no better than a coin flip. So testing in this scenario is NOT warranted because data that is no better than a coin flip is not data — it’s random chance.
However, it gets worse, much worse. PCR tests and antigen tests actually have no where near a 99% specificity level in practice, for various reasons. The peer-reviewed publication, Lee 2020 “Testing for SARS-CoV-2 in cellular components by routine nested RT-PCR followed by DNA sequencing,” performed a detailed analysis of the CDC PCR test, which was widely used in the first months of the pandemic, and found it had a 70% specificity (i.e. 30% false positives) and 80% sensitivity (20% false negatives). If we use these figures in the calculator, we get a catastrophic 30 out of 31 false positives. In other words, just one out of 31 positive test results is actually a real positive. And, again, we get zero false negatives.
This is a large part of why there have been so many allegedly asymptomatic carriers of the virus: 1) a “case” was defined as anyone who tested positive; 2) but with highly inaccurate tests and widespread testing of asymptomatics the large majority of “cases” are in fact false positives.
It is hard to overstate the importance of this understanding for the next pandemic.
Other policy changes that should also be enacted:
4. Don’t require that the new illness automatically be listed as the underlying cause of death
The National Vital Statistics System (NVSS) is part of the CDC and is responsible, as the name suggests, for tracking things like mortality data. NVSS issued a number of reports in March and April of 2020 that significantly changed guidance on how COVID-19 deaths should be reported in the U.S.
Specifically, the guidelines required listing COVID-19 in Part I of the death certificate as a “cause of death” even if COVID-19 was only suspected as being a contributing factor.
Not even a positive test for the virus was required. And the virus was not required to be the primary cause of death — it was enough to have “contributed to death,” with or without any testing, to list it in Part I. An NVSS report from March 24 stated (emphasis in original):
COVID-19 should be reported on the death certificate for all decedents where the disease caused or is assumed to have caused or contributed to death.
The March 24 report also stated that “the rules for coding and selection of the underlying cause of death are expected to result in COVID-19 being the underlying cause more often than not.”
And this report adds a key statement: “If the death certificate reports terms such as ‘probable COVID-19’ or ‘likely COVID-19,’ these terms would be assigned the new ICD code. It is not likely that NCHS will follow up on these cases.” In other words, there will be no review or accountability by NCHS for reporting Covid-19 as the cause of death even when there is doubt about the actual cause of death.
In a follow up report in April of 2020, NVSS recommended that Covid-19 be listed as the “underlying cause of death” on the death certificate if it was thought to be part of the causal chain leading to death:
If COVID–19 played a role in the death, this condition should be specified on the death certificate. In many cases, it is likely that it will be the UCOD, as it can lead to various life-threatening conditions, such as pneumonia and acute respiratory distress syndrome (ARDS). In these cases, COVID–19 should be reported on the lowest line used in Part I with the other conditions to which it gave rise listed on the lines above it.
CDC also required, when it coded death certificates (coding is double checking the accuracy of death certificates), that Covid-19 always appeared as the underlying cause of death unless it was listed only as a “contributing factor” in part 2 of the death certificate.
CDC should allow medical professionals to do their job for any future pandemic, and to list the new medical condition appropriately, given their training and on-the-ground experience. This practice will avoid “reassignment” of normally occurring deaths as due to the new ailment, which is very likely what happened for a significant portion of Covid-19 deaths.
5. Don’t include any association whatsoever as a “ — - death” unless there is deemed to be a causal connection between the illness and the death.
“If you died of a clear alternate cause, but you had Covid at the same time, it’s still listed as a Covid death. Everyone who is listed as a Covid death, doesn’t mean that was the cause of the death, but they had Covid at the time of death,” Illinois’s director of public health, Dr. Ngozi Ezeke, explained to reporters in April of 2020, early in the pandemic.
She expanded on these definitions as follows:
“I just want to be clear in terms of the definition of ‘people dying of Covid.’ The case definition is very simplistic. It means, at the time of death, it was a Covid positive diagnosis. That means, that if you were in hospice and had already been given a few weeks to live, and then you also were found to have Covid, that would be counted as a Covid death.”
This is the approach used for tracking all U.S. Covid-19 deaths because each state tallies the death statistics in much the same manner, counting a Covid-19 death as any death associated with Covid-19 in any way, regardless of the actual cause of death. CDC’s provisional mortality data describes the Covid-19 category as “all deaths involving Covid-19,” specifically not using language suggesting a causal linkage but only association.
This loose approach to deaths tracking is summed up well by Dr. Deborah Birx, the former White House Coronavirus Response Coordinator, in her statement from mid-2020 that “if someone dies with Covid-19 we are counting that” as a Covid-19 death.
For the next pandemic, it will be important to return to more established deaths reporting practices and to define a “____ death” as only those deaths that occurred as a direct result of the new illness, not in any loose association, or as a result of a lab test result only.
