The “false positive paradox” and why we shouldn’t test asymptomatic people

This is an academic paper that I and two co-authors wrote originally for the British Medical Journal after being invited to submit by a senior editor there. Unfortunately, our submission was rejected with no explanation and no peer review. We are continuing to submit to other medical journals and I’ll update this page on our progress.

Tam Hunt, J.D.[1]

Blaine Williams, M.D.[2]

Daniel Howard, Ph.D[3]

1 Univ. of California, Santa Barbara

2 Kaiser Permanente Oahu

3 Independent Researcher

Correspondence to:

Tam Hunt

tam.hunt@psych.ucsb.edu

Abstract

Widespread screening of asymptomatic people leads to high numbers of false positives when background prevalence is low, even with accurate tests. During the Covid-19 pandemic, not only has the background prevalence been low (vaccine clinical trial baseline testing finds 0.5–0.6% even during periods of higher prevalence), but the various COVID-19 tests are not very accurate. When inaccurate tests are combined with a low background prevalence, this results in a massive and unacknowledged problem of far more false positive test results than true positive test results, leading also to inaccurate characterization of COVID-19 hospitalizations and deaths.

It is well-known that widespread testing of people with a low probability of having the disease at issue will lead to high levels of false positives, even with accurate tests (Skittrall et al. 2020; Bokhorst et al. 2012; Dinnes et al. 2021; Madrigal et al. 2020). This has been described as the “false positive paradox” (Flender 2019). It’s a paradox because even quite accurate tests can lead to high levels of false positives when used widely in a population with low actual prevalence of a given disease.

For example, Skittrall et al. 2020 calculated that hypothetically screening 100,000 people chosen randomly from the general UK population in June 2020 would result in 25 times more false positives than true positives (50 false positives and 2 true positives), even with a test thought to have a very high 99.95% specificity.

Widespread screening during previous outbreaks and pandemics has generally not been recommended because of the potential for high false positives. The Center for Disease Control (CDC)’s 2004 guidance from the SARS pandemic (CDC 2004), for example, stated: “To decrease the possibility of a false-positive result, testing should be limited to patients with a high index of suspicion for having SARS-CoV disease.”

The debate around screening asymptomatic individuals

The World Health Organization (WHO) and CDC did, however, recommend testing of asymptomatic people early in the COVID-19 pandemic, but the CDC revised this guidance in August of 2020 to recommend not testing asymptomatics even after potential exposure, only to reverse course again after public and expert pushback in the U.S. (Fang, M. 2020).

The US Food and Drug Administration (FDA) issued a strongly worded letter to healthcare providers in November 2020 warning about the potential for false positives from antigen testing, describing the problems associated with screening populations with a low background prevalence of COVID-19 (FDA 2020). The letter reminds practitioners that: “As disease prevalence decreases, the percent of test results that are false positives increase.”

CDC’s most recent (March 2021) guidance does, however, still recommend widespread screening, which necessarily includes testing of mostly asymptomatics, despite the widely known issues regarding such policies. CDC’s guidance states: “Rapid, point-of care serial screening can identify asymptomatic cases and help interrupt SARS-CoV-2 transmission. This is especially important when community risk or transmission levels are substantial or high.”

Many countries, including the UK and the U.S., have engaged in widespread population testing (Mercer and Salit 2021: COVID-19 testing is the “largest global testing programme in history, in which hundreds of millions of individuals have been tested to date.”).

Not surprisingly, there appears to have been significant internal debate about this important issue in these agencies. Those arguing for screening asymptomatic people seem to have overlooked the extremely high false positive rate that such testing necessarily entails with low background prevalence (Skittrall et al. 2020; Dinnes et al. 2021).

We explain below why it is almost always unwise to test asymptomatics on a widespread basis. Such testing can lead to extremely high levels of false positives even with highly accurate tests. Unfortunately, the available PCR and antigen tests are not very accurate. And inaccurate tests combined with widespread testing of asymptomatics can lead to catastrophically high levels of false positives, as explained below.

The false positive paradox regarding prostate cancer

Let’s first, as an illustration of the issue, look at the history of screening for prostate cancer. In the U.S. screening for prostate cancer was normal practice, under the common sense notion that it is good to detect and treat illnesses early. What resulted, however, was a high level of false positives due to the false positive paradox, and a growing awareness that most prostate cancers don’t grow fast enough to be an issue for the patient, particularly in the elderly.

Consequently, the CDC, American Cancer Society, American Medical Association, and most other groups have stopped recommending widespread testing for prostate cancer, due to potential for false positives, overdiagnosis, overtreatment, and possible iatrogenic harm. CDC, for example, states at its prostate cancer website (CDC Prostate Cancer website, emphasis in original):

Possible Harm from Screening: False positive test results: This occurs when a man has an abnormal PSA test but does not have prostate cancer. False positive test results often lead to unnecessary tests, like a biopsy of the prostate. They may cause men to worry about their health. Older men are more likely to have false positive test results.

Possible Harms from Diagnosis: Screening finds prostate cancer in some men who would never have had symptoms from their cancer in their lifetime. Treatment of men who would not have had symptoms or died from prostate cancer can cause them to have complications from treatment, but not benefit from treatment. This is called overdiagnosis.

What is the background prevalence for COVID-19?

Even a test with a very high 99% specificity (1% chance of false positives), when used to screen asymptomatic populations with a low background rate of actual infection, will yield high levels of false positives.

The background rate of COVID-19 infection, even during high points, has always been relatively low. For example, Sadoff et al. 2021, the published results of the Johnson & Johnson vaccine clinical trial, including almost 40,000 participants in half a dozen countries, from late September 2020 to late January 2021, found a 0.5% PCR positive baseline (see Sadoff et al. 2021 supplementary appendix, p. 23).

Similarly, Baden et al., 2020, found a 0.6% background positive PCR test result in the 30,420 clinical trial participants for the Moderna vaccine, after initial testing. Study participants for this trial were selected based on being at higher risk for exposure to the virus and the testing was conducted from late July to late October 2020.

In the UK, the Government’s survey of the population in June 2020 found about 1 in 2,200 people with an active infection in the study window, which is 0.05%, an order of magnitude lower than the vaccine trials just mentioned (Connors and Williams 2020).

Voysey et al. 2021, the published results of the Astrazeneca vaccine trial, found 1.65% baseline antibody test positive results in their 20,675 study participants, but the antibody test (also known as serology testing) measures the presence of any past infection, not current infections, so this will necessarily be a significantly higher number than a snapshot in time of current infections, as is the case for PCR or antigen tests. The Astrazeneca trial study did not include PCR testing at baseline like the Moderna and J&J trials did.

Why we shouldn’t test asymptomatics for COVID-19

Common sense would suggest that a test with 99% specificity would return only about 1 in a 100 false positive results. But this is not how it works. The false positive rate is far higher when disease prevalence is as low as the studies just cited have found. In other words: the Positive Predictive Value of screening testing is very low when background prevalence is low (Bokhorst et al. 2012; Skittrall et al. 2020; Dinnes et al. 2021).

Here’s why: If we test 1,000 people randomly in a population where 1% have the illness at issue, and our test is 99% specific to that illness, we will have one true positive and one false positive for each 100 tests. So testing 1,000 people results in 10 true positives and 10 false positives.

Figure 1 below examines COVID-19 antigen testing, a type of test commonly used for screening, including primarily asymptomatics, since antigen tests became available in the summer of 2020. The figure is based on the British Medical Journal (BMJ)’s COVID-19 test accuracy interactive calculator (Watson 2020; see link for the calculator in the references; it’s educational to play with the calculator to see how different inputs affect false results, also shown in Figure 3).

In populating the three cells in the calculator (at the top of the image) we’ve conservatively assumed 1% pre-test probability of active infection, which is, based on the data reviewed above, a higher level of active infection than was found in the large vaccine clinical trials.

We also assumed 58% sensitivity and 99% specificity, which are the findings of a recent Cochrane meta-analysis combining 64 published studies of antigen test accuracy, when used to test asymptomatics (Dinnes, J. et al. 2021).

The result in this scenario is 50% false positives (1 true positive and 1 false positive) — even with a 99% specificity test. There would theoretically be zero false negatives, so the risk of missing actual infections is not at issue.

This result is not surprising because the numbers are the same as the hypothetical scenario just discussed. The Dinnes meta-analysis concludes similarly, but for a lower background prevalence and a slightly higher test specificity (99.6%): “At 0.5% prevalence applying the same tests in asymptomatic people would result in [Positive Predictive Value] of 11% to 28% meaning that between 7 in 10 and 9 in 10 positive results will be false positives, and between 1 in 2 and 1 in 3 cases will be missed.”

Figure 1. False positives account for 50% (1/2) of the positive results, even with a 99% sensitivity test when screening if the active infection level in the population is at 1%.

50% is the same as random chance. In other words, this 99% specificity test can do no better than a coin flip when declaring a positive result. So screening in this scenario is not warranted because data that is no better than a coin flip is not data — it’s random chance.

However, the situation is much worse than this because neither PCR nor antigen tests are close to a 99% specificity level in practice, for various reasons (Braunstein et al. 2021). Lee 2020 performed a lab analysis of the CDC PCR test accuracy, which was widely used in the first months of the pandemic, and found it had a 70% specificity (i.e. 30% false positives) and 80% sensitivity (20% false negatives). This level of inaccuracy matches the CDC’s own internal report that found 33% false results when its PCR test was released in late February 2020, as reported on by National Public Radio (Temple-Raston 2020).

Why intuition is a poor guide regarding testing

Intuitively, and in an emergency situation, we may think that a 70–80% accuracy rate is far from perfect but may still be “good enough.” But this is where common sense and intuition gets us — and the public — into trouble. If we input these figures in the BMJ calculator, we obtain a catastrophic 30 out of 31 false positives (Figure 2).

In other words, at a 1% pre-test probably (background prevalence), just one out of 31 positive test results is a true positive. And, again, we have zero false negatives, so the tests are not missing true positives in this scenario.

Figure 2. Using Lee 2020 findings regarding CDC’s PCR test inaccuracies when screening a population with active COVID-19 infection level of 1%.

This dynamic is a large part of why there have been so many allegedly asymptomatic carriers of the virus: 1) a “confirmed case” was defined by the CDC as anyone who tested positive (CDC Interim Case Definition 2020); 2) however, with highly inaccurate tests and widespread testing of asymptomatic individuals, the large majority of “cases” seem to have been false positives (Braunstein et al. 2021 makes a similar point).

Figure 3 summarizes the false results for the two scenarios already discussed, but in a range from 1–20% pre-test probability. False positives remain very high through 5% and higher background prevalence and false negatives remain low.

Figure 3. Summary of results: a) antigen test sensitivity of 58% and specificity of 99%; b) PCR test sensitivity of 80% and specificity of 70%. False positive results dominate in real-world low prevalence conditions.

a)

b)

The end result of these various policy mis-steps is fear of asymptomatic transmission leading to release of rushed and highly flawed tests (Bandler et al. 2020), which were used to screen widely in asymptomatic populations, which then seemed to find large numbers of asymptomatic carriers (who were actually in the large majority of cases false positives), creating a vicious cycle of faulty data.

This problem relates to more than just misidentifying positive COVID-19 cases; it also is relevant to data on hospitalizations and death rates. After testing became widely available, it became standard practice to test all patients admitted to hospitals in the U.S., regardless of symptoms. While this may have been a necessarily cautious step in order to minimize outbreaks in hospitals, it significantly inflated hospitalizations and deaths attributed to COVID-19. A positive test result was the primary basis for defining COVID-19 hospitalizations and deaths since no symptoms were required to designate a COVID-19 hospitalization or death as such.

In other words, since the CDC and WHO case definitions took the unprecedented step of defining a “confirmed case” as simply a positive lab test result, and then most jurisdictions also defined a COVID-19 hospitalization and a COVID-19 death in the same manner, if the large majority of positive test results are false positives, it is necessary to re-examine the pandemic surveillance data chain from the beginning.

We’ll close with a homework problem for the reader: think through how to apply this Bayesian thinking to the J&J baseline PCR testing, which as mentioned above, found 0.5% baseline PCR test positives at the beginning of the trial period. How many of those positive results were true positives? How many were false positives?

References

1. Baden LR, El Sahly HM, Essink B, Kotloff K, Frey S, Novak R, et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med. 2021;384(5):403–16.

2. Bandler J et al. “Inside the fall of the CDC.” ProPublica. 2020 Oct 15.

3. Bokhorst LP, Zhu X, Bul M, Bangma CH, Schröder FH, Roobol MJ. Positive predictive value of prostate biopsy indicated by prostate-specific-antigen-based prostate cancer screening: trends over time in a European randomized trial*. BJU International. 2012;110(11):1654–60.

4. Braunstein GD, Schwartz L, Hymel P, Fielding J. False Positive Results With SARS-CoV-2 RT-PCR Tests and How to Evaluate a RT-PCR-Positive Test for the Possibility of a False Positive Result. Journal of Occupational & Environmental Medicine. 2021;63(3).

5. CDC 2004, Supplement F: Laboratory Guidance — CDC [Internet]. CDC Notice regarding 2003 SARS. [cited 2021Jun15]. Available from: https://www.cdc.gov/sars/guidance/f-lab/downloads/f-lab-full.pdf

6. CDC Coronavirus Disease 2019 (COVID-19)2020 Interim Case Definition, Approved April 5, 2020 [Internet]. CDC Interim Case Definition. CDC; 2020 [cited 2021Jun15]. Available from: https://wwwn.cdc.gov/nndss/conditions/coronavirus-disease-2019-covid-19/case-definition/2020/

7. CDC Prostate Cancer website. What Are the Benefits and Harms of Screening for Prostate Cancer? [Internet]. CDC Prostate Cancer Screening. CDC; 2020 [cited 2021Jun15]. Available from: https://www.cdc.gov/cancer/prostate/basic_info/benefits-harms.htm.

8. Connors E, Williams C. Coronavirus (COVID-19) Infection Survey pilot: 2 July 2020 [Internet]. Coronavirus (COVID-19) Infection Survey pilot — Office for National Statistics. Office for National Statistics; 2020 [cited 2021Jun15]. Available from: https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/coronaviruscovid19infectionsurveypilot/2july2020

9. Dinnes_J, Deeks_JJ, Berhane_S, Taylor_M, Adriano_A, Davenport_C, Dittrich_S, Emperador_D, Takwoingi_Y, Cunningham_J, Beese_S, Domen_J, Dretzke_J, Ferrante di Ru(ano_L, Harris_IM, Price_MJ, Taylor-Phillips_S, Hoo-_L, Leeflang_MMG, McInnes_MDF, Spijker_R, Van den Bruel_A, Cochrane COVID-19 Diagnostic Test Accuracy Group. Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database of Systematic Reviews 2021, Issue 3. Art. No.: CD013705.

10. Fang M. CDC Quietly Changes Testing Guidelines To Exclude People With No Symptoms [Internet]. HuffPost. HuffPost; 2020 [cited 2021Jun15]. Available from: https://www.huffpost.com/entry/cdc-testing-guidelines-coronavirus-no-symptoms_n_5f4667fac5b64f17e136efd2

11. FDA 2020. Potential for False Positive Results with SARS-CoV-2 Antigen Tests [Internet]. U.S. Food and Drug Administration. FDA; [cited 2021Jun15]. Available from: https://www.fda.gov/medical-devices/letters-health-care-providers/potential-false-positive-results-antigen-tests-rapid-detection-sars-cov-2-letter-clinical-laboratory?utm_medium=email&utm_source=govdelivery

12. Flender S. The false positive paradox [Internet]. Medium. Towards Data Science; 2019 [cited 2021Jun15]. Available from: https://towardsdatascience.com/the-false-positive-paradox-f86448a524bc

13. Lee, S. Testing for SARS-CoV-2 in cellular components by routine nested RT-PCR followed by DNA sequencing. International Journal of Geriatrics and Rehabilitation 2(1):69- 96, July 17, 2020.

14. Madrigal A, Meyer R. Why Trump’s Rapid-Testing Plan Worries Scientists [Internet]. The Atlantic. Atlantic Media Company; 2020 [cited 2021Jun21]. Available from: https://www.theatlantic.com/health/archive/2020/10/do-rapid-antigen-tests-have-accuracy-problem/616681/

15. Mercer TR, Salit M. Testing at scale during the COVID-19 pandemic. Nature Reviews Genetics. 2021 May 4.

16. Sadoff J, Gray G, Vandebosch A, Cárdenas V, Shukarev G, Grinsztejn B, Goepfert PA, Truyers C, Fennema H, Spiessens B, Offergeld K, Scheper G, Taylor KL, Robb ML, Treanor J, Barouch DH, Stoddard J, Ryser MF, Marovich MA, Neuzil KM, Corey L, Cauwenberghs N, Tanner T, Hardt K, Ruiz-Guiñazú J, Le Gars M, Schuitemaker H, Van Hoof J, Struyf F, Douoguih M; ENSEMBLE Study Group. Safety and Efficacy of Single-Dose Ad26.COV2.S Vaccine against Covid-19. N Engl J Med. 2021 Jun 10;384(23):2187–2201. doi: 10.1056/NEJMoa2101544. Epub 2021 Apr 21. PMID: 33882225.

17. Skittrall JP, Fortune MD, Jalal H, Zhang H, Enoch DA, Brown NM, et al. Diagnostic tool or screening programme? Asymptomatic testing for SARS-CoV-2 needs clear goals and protocols. The Lancet Regional Health — Europe. 2021;1:100002.

18. Temple-Raston D. CDC Report: Officials Knew Coronavirus Test Was Flawed But Released It Anyway [Internet]. NPR. NPR; 2020 [cited 2021Jun15]. Available from: https://www.npr.org/2020/11/06/929078678/cdc-report-officials-knew-coronavirus-test-was-flawed-but-released-it-anyway

19. Voysey M, Clemens SAC, Madhi SA, Weckx LY, Folegatti PM, Aley PK, Angus B, Baillie VL, Barnabas SL, Bhorat QE, Bibi S, Briner C, Cicconi P, Collins AM, Colin-Jones R, Cutland CL, Darton TC, Dheda K, Duncan CJA, Emary KRW, Ewer KJ, Fairlie L, Faust SN, Feng S, Ferreira DM, Finn A, Goodman AL, Green CM, Green CA, Heath PT, Hill C, Hill H, Hirsch I, Hodgson SHC, Izu A, Jackson S, Jenkin D, Joe CCD, Kerridge S, Koen A, Kwatra G, Lazarus R, Lawrie AM, Lelliott A, Libri V, Lillie PJ, Mallory R, Mendes AVA, Milan EP, Minassian AM, McGregor A, Morrison H, Mujadidi YF, Nana A, O’Reilly PJ, Padayachee SD, Pittella A, Plested E, Pollock KM, Ramasamy MN, Rhead S, Schwarzbold AV, Singh N, Smith A, Song R, Snape MD, Sprinz E, Sutherland RK, Tarrant R, Thomson EC, Török ME, Toshner M, Turner DPJ, Vekemans J, Villafana TL, Watson MEE, Williams CJ, Douglas AD, Hill AVS, Lambe T, Gilbert SC, Pollard AJ; Oxford COVID Vaccine Trial Group. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK. Lancet. 2021 Jan 9;397(10269):99–111. doi: 10.1016/S0140–6736(20)32661–1. Epub 2020 Dec 8. Erratum in: Lancet. 2021 Jan 9;397(10269):98. PMID: 33306989; PMCID: PMC7723445.

20. Watson J, Whiting PF, Brush JE. Interpreting a covid-19 test result. BMJ. 2020;:m1808. Online at https://www.bmj.com/content/369/bmj.m1808/infographic.