FDA fast-tracks COVID-19 tests, but it’s unclear how accurate they are

Kendra Boroff believes she contracted the coronavirus on her 71st birthday, February 20, when her family went out for a celebratory dinner, perhaps from their waiter, who was coughing into his elbow. Four days later, she developed a fever and a raging sore throat.

“You feel like you’re suffocating,” recalled Boroff, a real estate agent in Maineville, Ohio. “You cough and breathe with the top fourth or maybe less of your chest, because everything else is in a vise.”

Over the course of the next three weeks, as Boroff started getting chills and nausea, a series of doctors would suggest that it could be the common cold, bronchitis, or pneumonia. She tested negative for the flu, and her chest X-rays showed signs of lung damage, including white patches called “ground-glass opacities” that are common in COVID-19 cases. By March 7, she was pretty sure it was COVID-19, but she couldn’t get a test until she arrived at the emergency room at the University of Cincinnati Health Center on March 19. She had a 103 degree fever and her oxygen levels were plummeting, so the doctors admitted her immediately.

Nearly a week later, as Boroff’s condition was stabilizing, the test results came back: negative.

Boroff was flummoxed, but her physician was clear that she had the virus, no matter what her test said.

“‘This is my diagnosis,’” she recalled him saying. “‘There is no other explanation.’”

Tests turning up negative even when all signs point to COVID-19 has been a common experience in American hospitals over the past month, public health experts have told ProPublica. It’s unclear what proportion of these negative results are inaccurate—known as “false negatives”—and whether that’s due to some external factor, like bad sample collection, or because of an issue inherent in the tests’ design.

Neither the major test manufacturers, the US Food and Drug Administration, nor the US Centers for Disease Control and Prevention would say how common false negatives are. While the FDA requires test-makers to report any known instances of false negatives as a condition of granting them provisional approval, known as emergency use authorizations, no such reports are visible in a database the agency maintains for that purpose.

Without much data on how COVID-19 tests are performing in the real world, concerns are mounting that a lack of accurate testing will make it more difficult for America to relax social distancing, as the ability to track and trace new infections will be critical for any strategy to reopen the country.

Those warnings have reached Capitol Hill, where Texas Democratic Rep. Lloyd Doggett had heard from a doctor in his district about the accuracy of the tests. On Thursday, he and Rep. Rosa DeLauro of Connecticut sent a letter to the FDA demanding more data about the prevalence of false negatives in both the diagnostic tests currently in widespread use, as well as inaccuracies in the coming wave of rapid blood tests that detect immunity once the infection has passed.

“I’m very concerned about it,” Doggett told ProPublica. Too many false results, he worries, could lead to a new surge of infections when people go back to work or are allowed to gather in bars, sports arenas, and restaurants. “They have to monitor this very closely to ensure that we’re not creating false expectations, and in the process ending up with an epidemic that is even worse than the one we have now.”

Lowering the bar in an emergency

In the early days of the pandemic, the FDA, which regulates diagnostic tests, was criticized for not moving quickly enough to make testing widely available. For much of February, the only available test was the CDC’s, which initially had flaws when it was sent out to public health labs. Only on Feb. 29 did the FDA announce a new policy that made it easier for private labs and academic medical centers to make tests available as well.

Since then, the ongoing need for even more testing capacity across the United States has pushed the agency to loosen its typical requirements for manufacturers to prove that their tests are accurate before allowing them onto the market.

Normally, to get FDA approval, diagnostic makers need to run trials to gather evidence on their tests’ performance, a process that can take months or even years. The agency is currently skipping a lot of those steps by issuing emergency use authorizations.

Manufacturers are now required to run their COVID-19 tests on a minimum of 30 positive samples and 30 negative samples. They must demonstrate to the agency that the test has at least a 95-percent sensitivity, meaning it must correctly identify at least 95 percent of the positive samples as having the coronavirus, and 100 percent specificity, meaning that it must accurately identify all the negative samples as not having the coronavirus.

But the manufacturers are demonstrating their diagnostics’ performance with what’s known as “contrived samples,” which are not taken from actual patients. A contrived sample is made by taking coronavirus RNA made in a lab and putting it into a medium that mimics nasal mucus.

“This is supposed to represent a swab specimen, but it’s not a positive sample from a real patient, and that does make a real difference,” said Benjamin Pinsky, medical director of the Clinical Virology Laboratory for Stanford Health Care.

It’s not clear if the concentrations of virus on the simulated samples are representative of the full range of material taken from patients’ bodies in the real world. Pinksy says that it’s reasonable for the FDA to allow the use of contrived samples, because it makes it much faster for a manufacturer to run validation studies, and the need for speed has been pressing.

“But then we need to have studies to compare these assays and see how they perform with real-world samples, and whether some are more or less sensitive and whether some are more or less specific,” Pinsky said. “We don’t know the answer to these questions at this point.”

To compensate for the lower standard up front, experts say the FDA should track data on accuracy to make sure the tests are performing as expected, but this is easier said than done.

“In diagnostic tests in particular, it’s very difficult to know if something is failing,” said Alberto Gutierrez, former director of the FDA’s Office of In Vitro Diagnostics and Radiological Health. “When are you getting more erroneous results than you should? It’s not always easy to figure out.”

Swiss manufacturer Roche, whose test was authorized by the FDA on March 12, told ProPublica it couldn’t give specific numbers about its test’s actual rate of false negatives and false positives, though it said studies have demonstrated its test could detect very low levels of the coronavirus.

“Clinical studies, which take months to run and would be part of a regular (nonemergency) test approval process, are needed to give us an exact percentage of false negatives and false positives,” Roche spokesman Mike Weist wrote in an email. “We will continue to work with the FDA on ongoing studies post-EUA that will allow us to potentially say more in the future.”

Abbott, which makes a rapid COVID-19 test, also said that “performance characteristics, including accuracy data, will continue to be collected in the field.”

Abbott and the testing firms LabCorp and Quest Diagnostics all told ProPublica that tests should be used by physicians along with other information to form a diagnosis.

Even good tests can give inaccurate results

Clinicians and researchers said that a number of factors could cause inaccurate results on COVID-19 tests, and many of them have nothing to do with the test’s design.

For starters, the timing of when a patient receives the test matters. “If you’re far out from the initial exposure, the more days you are after onset, viral load goes down,” explained Stanford’s Pinsky. Viral load refers to the amount of virus that is being emitted from an infected person’s cells, and if that drops too low, even a person who still has an active infection may test negative.

Another issue is where the virus is in a person’s body. As the disease progresses, scientists think the virus tends to move down into a patient’s lungs, so the window of time when a nose swab will return a positive result may be limited.

“One of the issues with this stupid virus is that if it’s down in your lungs, and we’re putting a swab up your nose, that’s not the best way to measure what’s in your lungs,” said Alex Greninger, assistant director of the clinical virology lab at the University of Washington Medical Center.

While it is possible to stick a scope down a patient’s airway to collect a sample from the bottom of the lungs, this is a much more complex procedure that requires sedating the patient. Technicians can ask a patient to cough up phlegm, known as sputum, but doing so substantially raises the risk of infecting health care workers. Even with a nasopharyngeal sample collected with a nose swab, one needs to collect it properly, which involves sticking the swab quite far up a patient’s nose.

Daniel Brook, a freelance journalist and historian in New Orleans, says he thinks his test result may have been a false negative because he was incorrectly swabbed.

During Mardi Gras, he hung out with a friend who was visiting from Manhattan. A few days later, as he started to get night sweats and chills, Brook’s friend texted to say that he had tested positive for COVID-19. Brook has asthma, so when he started to have trouble breathing, he went to an urgent care center, which said it didn’t have enough tests to give him one.

Four days later, as Brook found himself even more winded going up stairs, he and his girlfriend, who also had symptoms, received a letter from an emergency room doctor that would get them a test at a drive-through center. They first were tested for the flu and then finally for COVID-19.

“This flu test was way the hell in there. It was almost like you ate too much hot pepper,” Brook said. “And then we had this COVID test, and it was barely in the nose at all, which may be one of the issues.” Nine days later, they received their results: both were negative.

Brook was confused. He had been trying to tell all of the people he had been in contact with, like his barber, that they might have been exposed, and he shared the good news with many of them. But his doctor told him that clinically, he had all the symptoms of COVID-19 and that his diagnosis would not change based on his test result.

Even if the sample is taken correctly, mishandling of the swab can also invalidate the result. RNA is similar to DNA but due to chemical differences is a much more fragile material and degrades more readily. This coronavirus is an RNA virus, essentially a string of RNA encased in a membrane “envelope.”

Abbott, one of the test makers, said that it recommends that samples be kept for no more than eight hours at about 60 to 85 degrees Fahrenheit, or refrigerated for 72 hours. “People should make sure it is tested in a timely fashion,” Abbott said in its statement to ProPublica.

None of this bodes well for the numerous labs that have reported backlogs of tens of thousands of samples that are waiting to be tested.

A technician at an academic laboratory, who asked for anonymity because he is not authorized to speak on behalf of his university, described seeing basic mishandling of samples that is probably ruining dozens of patients’ test results.

“I don’t know why, but with COVID, we’ve just been awash with problems,” he told ProPublica. “Even simple things like caps not screwed on tightly—we’ll get a bag of samples, and two or three of them will be leaking, so you have this media completely soaking the inside of the bag. If one of those leaking samples is positive, you’ll have droplets all over the bag.”

Those samples, the technician said, often can’t be processed at all. His experience isn’t unique: in Alabama in late March, hundreds of samples were ruined in transit to a lab in Montgomery.

The dangers of inaccuracy

In the absence of data, physicians and public health officials are left to guess how many false negatives may be occurring—which could have serious consequences both for individuals and for combating the spread of the disease.

“You want to be right every time, because you miss somebody, and tell them that they’re negative, then you’re infecting people,” said Gutierrez, the former FDA official. “Let’s say you consider Amazon essential, and at the warehouse they’re testing people, even if they miss one or five people out of 100, that can be problematic.”

In addition, false negatives can make it more difficult to track spread of the virus, since those patients are not reported as confirmed cases and people who die without a positive test result won’t be counted in COVID-19 mortality statistics.

False positives also present problems. If you mistakenly think a patient has COVID-19, “then you have the potential to clog up the health care system and waste personal protective equipment and the time and effort of health care workers who think they are caring for individuals with COVID-19,” Stanford’s Pinsky said. “In addition, you’re producing a lot of anxiety for the patient.”

Pinsky says he hopes that real-world data will be gathered on the tests’ performance, especially as more and more come on the market: “If physicians have this information, they could move on to a different, better-performing test and use that instead.”

Dr. Yukari Manabe, associate director of Global Health Research and Innovation at Johns Hopkins Medicine, estimates that 10 to 25 percent of test results are false negatives. That’s not based on any data, she cautions, since hard evidence isn’t available. But she has been noticing many patients in the Hopkins system being tested more than once, when the first result doesn’t match their clinical symptoms.

Like others, Manabe acknowledges that the FDA has needed to green-light tests quickly in order to get them out into the public. But she laments that companies weren’t encouraged to develop diagnostics earlier, which might have allowed the agency to keep the bar for approval higher and also churn out more tests sooner.

“If people had seen the writing on the wall back in December, someone should’ve paid these companies what they needed to develop these tests on platforms that could’ve been rapidly ramped up to millions of tests,” Manabe said.

Instead, a test shortage caused doctors to limit tests to only the sickest patients, at a time when the virus had probably moved out of the back of the nasal cavity and into their lungs. A larger supply would have allowed for testing more people as soon as they started showing symptoms. That would have resulted in a lower rate of false negatives, Manabe said, since nose swabs are more likely to detect the virus soon after it has been contracted.

The next wave of tests may be even less accurate

The questions swirling around the accuracy of the COVID-19 diagnostic tests are likely to persist as the next set of tests—antibody blood tests—start hitting the market. Already, the FDA has authorized the first of these tests, which search for molecules in a patient’s blood that can indicate if the immune system did battle with the coronavirus. Unlike the swab-based tests, which look for the viral RNA that indicate active infection, antibody tests are used to seek evidence of a past encounter with the virus.

Antibody tests are already seen as a critical tool in lifting lock-down measures, because they could potentially be used to figure out who has immunity to the coronavirus. In this case, false positives would be the greater concern, because it could be dangerous to tell someone that they have antibodies and are safe to go back to work when that is a false signal.

There are issues that need to be figured out before rushing to rely on these tests, Stanford’s Pinsky warned. What level of antibodies are needed to mean that someone is protected? And if you are protected, how long are you protected? The answers to these basic questions are still unknown, he said. This week, the World Health Organization put out guidance recommending against using antibody tests for clinical decision making.

The FDA, meanwhile, is lowering the bar even further. On March 16, it issued new guidance allowing manufacturers to distribute tests even before receiving emergency use authorization, for a “reasonable period of time”—about 15 days—after a diagnostic maker had validated the test internally and while preparing its request to the agency for an EUA.

Local governments are desperate enough for tests that they’ll buy them without assurances of accuracy at all. Chicago recently ordered 11,000 antibody tests made in South Korea that had not been reviewed by the FDA but are legal to distribute as long as they include several disclaimers including a recommendation that any negative result be confirmed with a diagnostic test.

“There’s no time really to put the effort into saying, ’Where’s the problem here?’” said Catherine Troisi, an epidemiologist at the University of Texas Health Science Center. “I’m not saying the test is bad. But what good is a test if you don’t know it’s giving you reliable results? We just don’t know.”

Promoted Comments

• ColdWetDog Ars Praefectus et Subscriptor
  Totality wrote:

  What’s the accuracy rate of other (non-coronavirus) tests?

  The obvious answer is that ‘it depends’. There are tens of thousands of ‘other tests’ but I’m going to presume you mean ‘viral detection tests’ for the purposes of this discussion.

  It still depends. Different tests have different false positive and false negative values. The importance of those numbers depends on the clinical situation. Let’s use the routine strep test. The ‘rapid strep’ test is a linked antibody to a streptococcus antigen. The antibody is stuck to the paper and rigged such that when it picks up the antigen, it changes in such a way to make a visible line on the paper. It’s about 95% sensitive and 99% specific. If it’s done correctly – that’s another issue.

  So, five times out of a hundred tests, if all is done per the manual, you will have a patient that does have strep in their throat but you gave them a pass. The next question is what are the implications for that? Turns out that for strep throat, missing the diagnosis is a nothing burger. We used to think that it could create all sorts of havoc but that is extremely rare. Much rarer than side effects from the antibiotic. So, the downside for missing strep throat isn’t all that much.

  Now, lets switch to HIV or Ebola. The downsides for a missed diagnosis are much higher. You want a test that is both sensitive and specific since the downsides for an false positive diagnosis are also pretty high. Couple of ways to deal with that. You could try to create a single test with great sensitivity and specificity. Perhaps not technically possible. You can use multiple types of tests to create independent statistical values that much improves your diagnosis. At the expense of time and money and complexity (another way for errors to creep in). For HIV it was suggested that any positive test get a backup, more complicated test and that negative tests be repeated after a couple of weeks if there was any clinical suspicion of infection (as opposed to simply screening).

  At present, the availability of tests is a huge issue. Hard enough to test people once, much less twice or three times.

  It’s rather complex and you can spend a lot of quality time in Masters of Public Health or similar programs going over this stuff.

• PedroDaGr8 Smack-Fu Master, in training
  ColdWetDog wrote:

  show nested quotes

  Now, lets switch to HIV or Ebola. The downsides for a missed diagnosis are much higher. You want a test that is both sensitive and specific since the downsides for an false positive diagnosis are also pretty high. Couple of ways to deal with that. You could try to create a single test with great sensitivity and specificity. Perhaps not technically possible. You can use multiple types of tests to create independent statistical values that much improves your diagnosis. At the expense of time and money and complexity (another way for errors to creep in). For HIV it was suggested that any positive test get a backup, more complicated test and that negative tests be repeated after a couple of weeks if there was any clinical suspicion of infection (as opposed to simply screening).

  I can actually speak to HIV a bit more in depth. The company I work for manufactures diagnostic assays used to screen blood donations for HIV and several other diseases. You are correct that a diagnosis requires a confirmatory assay but this is far from the only safeguard. For screening assays, the assays are actually designed to be biased against false negatives. Which means, false positives do occur and are accepted if they are limited in scale. All positive result samples are retested on the same assay and if they repeat positive the donor can never donate blood again. Note that this is not confirmatory testing because the same assay is used. This is just to eliminate any “one-off” discordant results. Irrespective of the results of the confirmatory assay, that person would be banned from donating blood permanently.