[Skip to second paragraph if you are not interested in the German context of the false positives issue.]
On June 5, 2020, OVALMedia’s Robert Cibis interviewed the Austrian microbiologist and infectious disease specialist Dr. Martin Haditsch about laboratory tests and specifically the PCR test that is used globally to detect the Sars-CoV-2 virus in a person. The interview [in German] broached the issue of false-positive test results in the context of a low-prevalence disease and imperfect tests. Two Youtube copies of the one-hour interview have a total of over 100,000 views at the time of writing. The next day, Swiss entrepreneur and Youtuber Samuel Eckert presented a 20-minute summary and explanation of the false-positive issue using an interactive Excel spreadsheet. His video currently boasts over 225,000 views with 15,000 likes. Possibly in response, the German Federal Minister of Health Jens Spahn, a banker by training, said in a brief interview contained in a tweet from public TV channel ARD on June 14 that if the COVID-19 prevalence continued to drop and testing was simultaneously expanded (as has been the case in many Western countries since mid-April) into the millions then you would eventually obtain more false-positive than correct-positive results.
Six weeks later, in a media briefing about the Province of Ontario’s safe reopening of schools, Associate Chief Medical Officer of Health Dr. Barbara Yaffe cautioned against seeing wide-spread COVID-19 testing as a solution. She went on to state that “in fact, if you’re testing in a population that doesn’t have very much COVID, you’ll get false positives almost half the time.”
In order to understand the impact of two test characteristics – sensitivity and specificity – we can use a confusion matrix to display true and false positive test results along with true and false negative test results. For example, in Geography we use confusion matrices to assess the accuracy of the classification of a remotely sensed image. The confusion matrix shows how many pixels of a certain land use class such as agricultural were correctly classified as agricultural or misclassified as another land use like forest. This is based on a ground-truthed image for comparison in a limited part of the study area. Similarly, the following spreadsheet estimates how often a positive or negative PCR test result is found in a true carrier of the virus vs an unaffected person.
The true proportions and counts of “infected” and not “infected” are based on the number of tests completed and the true prevalence of COVID-19 in the population, which is largely unknown. At the tail end of winter, coronaviruses are typically causing cold infections in some 5% to 10% of the population. As COVID-19 has tapered off over the summer, a value below 1% seems reasonable, here set to 0.5% as an example. In recent days, there were 40,000 or more tests completed in Ontario with some 400 new “cases” detected for a test-positivity rate of around 1%. Note that in conjunction with the PCR test, I like to put “cases” and “infected” in quotation marks since the test does not distinguish sick people carrying an infectious virus load from healthy, presymptomatic, or asymptomatic people carrying traces of inactive genetic material from the virus.
With a prevalence of e.g. 0.5%, we expect that 200 out of 40,000 tests are true positives. If we assume the sensitivity characteristic of the test at 99%, we get 198 correct positives out of the 200 true “infections” that should be detected, while two are missed. These two misses are false-negative results. False negatives are problematic, since potentially infectious persons are told that they don’t pose a risk to their environment. However, at a low prevalence of the disease, these misses are very small, even negligible, in comparison with the correctly identified negatives.
After subtracting 200 “infected” from the 40,000 tests completed, there are 39,800 left who should test as not “infected”. However, medical tests are usually imperfect in that they can both miss a condition present (false negatives, see above) as well as indicate the presence of a condition when it is not there (false positives). The characteristic that describes how accurate the test is in this latter respect is called specificity. It refers to how specific the test is geared towards its target, here the Sars-CoV-2 virus, rather than picking up other targets. The specificity of the PCR test for Sars-CoV-2 is a bit of a mystery and moving target, but before I discuss it, I will go through one example to explain the significance of false-positive results in the current phase of the pandemic.
With an assumed specificity of 99.5%, our test would determine 39,601 correct negatives out of 39,800 persons who are truly not “infected”. An issue arises with the remaining 199 false positives, which are wrongly detected by the test although they do not carry the virus. While the number is small in proportion to the correct negatives, we need to view it in relation to the 198 correct positives. From the perspective of each person testing positive, the chance that they are in fact falsely positive is 199 : 198, thus 50% of the 397 total positive test results are false positives, in line with the warnings by the officials cited above. The relationship is only this large due to the low prevalence of the disease. You can use the spreadsheet link above to examine the effect of increasing or decreasing the prevalence (try 0%!) and modifying the test characteristics.
Due to the significant restrictions imposed on test-positive persons’ lives, 50% false positives is a highly problematic proportion. In addition, a difference between 200 and 400 new “cases” may affect the assessment of the current public health threat, although the raw count of positive PCR tests is almost meaningless without considering who was tested and how many tests were completed, an issue which shall be discussed at another time.
Some critics of the pandemic response are using a lower specificity of 98.6% and come to the conclusion that all (!) positive test results in certain jurisdictions such as Germany are false positives and therefore the pandemic has ended. That values stems from a study of lab results (external PDF, see page 12) conducted by a German accreditation body in April, which found an average of 98.6% correct negative results across over 400 participating labs. Other experts however have noted that the actual proportions of positive test results has gone down to values as low as 0.6% in Germany, 0.3% in Canada, and even 0% in New Zealand, which – given the large numbers of tests completed – would not be possible if the false positives were any higher than these values. This can be explained by the fact that the quality assurance study reported the results for individual gene sequences but the testing protocols in some countries were modified to require testing of at least two gene sequences. In this case, the false positive rates of 0.5% for the E gene region (which I used in the above example) and e.g. around 2% for another characteristic gene would have to be multiplied, resulting in extremely low false-positives of 1 in 10,000 or so. However, Public Health Ontario’s in-house test methods target the E gene and clearly state that “Specimens with a single target detected … will be reported as COVID-19 virus detected, which is sufficient for laboratory confirmation of COVID-19 infection.” (emphasis added)
In summary, false-positive PCR test results likely are, or have been, an issue in some jurisdictions during some phases of the corona crisis, and politicians and health officials should be more transparent about this. Questions that the media should be asking include:
- When did you become aware of the impact of false positives?
- What is the magnitude of the problem currently?
- Have lab testing procedures been modified to address the issue?
- Is wide-spread testing still meaningful at this point in the pandemic?
In fact, just yesterday, the Ontario government changed course by discouraging asymptomatic people from getting tested. Yet this reversal seems more related to preserving lab capacity for symptomatic persons and those with suspected exposure, who need faster test results, than to the fundamental issues with over-testing.