In retrospect, it was probably a mistake to take the mask off. But what are you gonna do? David Flint and his wife had to check in on her father up in the Bronx. Flint’s wife is her father’s home health aide, and the older man had just come home from having foot surgery. For more than an hour, they were all sitting there around the bedside — Flint, his wife and his sister-in-law. It’s a long time to wear a mask when not everyone else is. “After a while, I took it off,” Flint said.
Flint was vaccinated against COVID-19 already, so he assumed he’d be immune. A social worker in New York who provides home hospice services to the dying, he was one of the first people in the country to get the vaccine. By the time he sat down in his father-in-law’s bedroom on Jan. 19, he’d been fully vaccinated for a week. The odds were in his favor.
But odds are fickle things. In a game of chance, not everybody gets to win, even if the odds of winning are high. Flint rolled … and he lost, diagnosed with a mostly asymptomatic case of COVID-19 on Jan. 25. That, by itself, wasn’t a shock. He’d known that some people would still get the virus despite being vaccinated. Even the mRNA vaccines’ famed “95 percent efficacy” was really a measure of how well the vaccines prevented symptomatic cases. But Flint didn’t expect to be one of the people who slipped through the cracks. More importantly, though, he expected somebody to care. “I thought there’d be some mechanism,” he said. But nobody asked him about his vaccine status when he got tested. There was nowhere to file that information with his doctor. And that was the part that confused Flint. “Shouldn’t somebody want to know?” he asked.
Yep, they should. And they do. Efforts are already underway to gather information that will help scientists understand how effective the COVID-19 vaccines are in the real world. But “How well do vaccines work?” and “Should we be counting every vaccinated individual who gets the disease?” are two different questions.
That complication starts with some basic facts about the effectiveness of the Pfizer and Moderna vaccines currently available in the U.S. Scientists say there is a difference between “efficacy” and “effectiveness.” Efficacy is the 95 percent number you get from a clinical trial. Effectiveness is what the number is once you’re vaccinating millions more people, some of whom will be older or sicker or more likely to be exposed to a virus than trial participants. It’s a metric that encompasses all the messiness of real life, including that vaccines won’t always be administered in ideal ways, said Dr. Kelly Moore, deputy director of the Immunization Action Coalition, an organization that works with the Centers for Disease Control and Prevention to educate the public on vaccines. “You have people who forget to come back for a second dose or come back late. Perhaps there’s a dosing administration error, or a storage problem,” she said. And that’s before you even start getting into whether new variants like the B.1.1.7 — originally found in the U.K. but expected to become dominant in the U.S. by March — might be more resistant to the vaccines than the variants those vaccines were tested against back in the fall.
The CDC will be tracking real-world COVID-19 vaccine effectiveness in multiple studies, using different methodologies in different places at different times. Some studies — like one that tracks groups of vaccinated and unvaccinated health-care personnel over time — are already underway.
Other studies are just getting off the ground. One CDC effort will piggyback on an existing system created to track the effectiveness of flu vaccines. At five medical research centers — in Michigan, Pennsylvania, Texas, Washington state and Wisconsin — every person who comes in with a cough or other respiratory symptoms can become a study participant. All of them will be tested for COVID-19. The ones who test positive are the cases; the ones who test negative become the controls. Researchers will then compare rates of vaccination between the two groups. Those studies are just beginning, though, because you can’t study the vaccine until people actually start getting it.
“It’s only as the vaccine gets rolled out to larger sections of the population that it becomes feasible to do. We’re just getting to that point now in Wisconsin,” said Dr. Ed Belongia, director of the Marshfield Clinic Research Institute’s Center for Clinical Epidemiology & Population Health — the flu vaccine-effectiveness research center in Wisconsin — on Feb. 11. “You can’t learn anything when only 1 percent of the population is vaccinated.”
The CDC is taking multiple approaches to this because the real world lacks something that’s easier to control in a clinical trial: randomization. Unlike in that lab setting, you can’t pick some people to get the vaccine while denying it to others. What’s more, people don’t just sign up for clinical trials randomly, and that affects the results. People who want to participate in a study may differ in some ways from the population as a whole. Doing different kinds of studies that compare groups in a variety of ways helps reduce some of the uncertainty in the overall results.
But none of these efforts will study vaccine effectiveness by counting all the individual cases like Flint’s. There is CDC research aimed at doing that, but it’s not about vaccine effectiveness. Instead, that project, a partnership with state health departments, is meant to spot trends in who the vaccine isn’t working for.
Those methods of tracking vaccine effectiveness aren’t new, even if the virus is. Scientists study post-introduction vaccine effectiveness for every new vaccine that comes out, said Dr. Katherine Fleming-Dutra, a member of the Vaccine Effectiveness & Evaluation Team in the CDC’s COVID-19 Response. And that research has proven crucial to disease prevention.
For example, as part of an effort to eradicate measles in the U.S., scientists began tracking the decades-old measles vaccine in the 1980s. The studies taught them that one dose of this vaccine wasn’t cutting it, according to Dr. Walter Orenstein, professor and associate director of the Emory Vaccine Center at Emory University. In 1989 the CDC and the American Academy of Pediatrics began recommending that everyone get two doses. If you weren’t looking closely, it would have been easy to miss that a second dose was necessary. The first dose of measles vaccine is 93 percent effective. But the disease spreads so easily and rapidly that 93 percent wasn’t quite good enough, Orenstein said. With the second dose, the vaccine becomes 97 percent effective at preventing measles.
The flu vaccine, meanwhile, goes through this process every year. That’s why there’s that network of research centers for the CDC to use to study COVID-19. That system turns out results on flu vaccine effectiveness twice a year, and preliminary results are able to be put together with as little as a month or two of data. But that doesn’t mean we’ll have results as quickly on the COVID-19 vaccines. As the entire public health system has taken great pains to explain this past year, the flu and COVID-19 are not the same beast — and no one I spoke to was willing to estimate how long the results for COVID-19 will take.
That’s because a number of complications will make it harder (and possibly take longer) to do the same kinds of studies for COVID-19. For example, the seasonal flu has, at this point, predictable annual checkpoints. Vaccines start rolling out in the fall. By December or January, as flu cases really begin to rise, everyone who is going to be vaccinated already has been, and the proportion of Americans who are vaccinated is roughly the same from year to year. With COVID-19, scientists are looking at a disease that has a high prevalence in some places and not in others as well as rolling out brand-new vaccines.
There are other challenges to tracking the effectiveness of COVID-19 vaccines, said Emily Martin, professor of epidemiology at the University of Michigan and co-director of the Michigan Influenza Center. The flu networks have long relied on people coming in to see a doctor about their respiratory symptoms. That’s where they get signed up for the studies. But the expansion of COVID-19 testing has taken place largely outside doctors’ offices, and those testing centers may or may not leave a record of negative (or positive) diagnoses.
When the data finally does come in, it’s likely to show us that David Flint is not alone in contracting COVID-19 after getting two doses of the vaccine. But these studies can also bring good news as well. That’s because the fun thing about vaccines is, “How well do they work?” isn’t just about individuals. For example, when researchers studied the real-world effectiveness of pneumococcal vaccines, they found that rates of the disease fell among older people even though it was only children who were being vaccinated. That’s because kids were the primary carriers of the disease. Once they stopped contracting it, so did their grandparents.
And vaccine effectiveness isn’t just about how many people test positive. One thing we’ve learned from flu vaccine studies is that the vaccine can reduce the severity of the disease, even if you do still contract it post-vaccination. These studies will help us figure out what’s going on with COVID-19 as well. Stuff like that matters. After all, when Flint got COVID-19, he just had a sore throat. His unvaccinated family members, though, were worse off.