What Does Vaccine Effectiveness Mean?
What Does Vaccine Effectiveness Mean? by Peter Dorman
When technical specialists adopt an everyday word, they often give it a meaning that differs from its everyday use. This can be misleading for nonspecialists, especially when little effort is made to explain the difference. A well-known example is “significance”, which means one thing when used in statistical work and another when it just denotes whether something is important.
Let’s look at another example, vaccine “effectiveness”. What do people most want to know about the coronavirus vaccines? How much protection they give you against the risk of getting infected with the virus, right? And how much protection they give against more severe symptoms, such as those requiring hospitalization or resulting in long Covid. When public health authorities throw out numbers about vaccine effectiveness, that’s probably how most people interpret them.
But that’s not what effectiveness means in medical research. When pharmaceutical companies or public health outfits conduct effectiveness tests, they assemble and compare two groups, a treatment and a control (or multiple treatment groups with different protocols). The treatment group gets the vaccine, the control group doesn’t. Who gets assigned to which group is determined randomly, and participants don’t know which one they’re in. (The controls get injected with a placebo.) Then they go about their life, monitored to see if they get infected or not. Vaccine effectiveness is a ratio, the fraction of the control group that gets infected divided by the corresponding fraction of the treatment group; it’s a ratio of two ratios. You can also calculate effectiveness within subgroups, like treatments-over-65 and controls-over-65. If the trial is conducted properly, the samples are representative and large and the public health context, including the virus variant, is stable, you can generalize effectiveness in the samples to the population as a whole.
Now notice a subtle difference in language. The everyday use of “effectiveness” is effectiveness against the virus. The research use is effectiveness relative to the control group. This is immense, but widely misunderstood and seldom explained.
Here’s a numerical example. Suppose a typical unvaccinated person going about life in a typical way faces a 1% risk of getting infected with Covid over the course of a month. Suppose also that a vaccine is introduced with 95% effectiveness, as that term is used in medical research. This means that a vaccinated person exposed to the same risk factors would have only a .05% chance of getting infected during the same time period.
Next, imagine that a new virus variant appears, combined with more relaxed public behavior—more indoor gathering, less masking. Let’s say that an unvaccinated person now has a 5% monthly risk of infection. If the vaccine is equally effective against the new variant, our typical vaccinated person now has a .25% risk of infection. The numerator and denominator have both risen fivefold, but the effectiveness ratio of treatment vs control is unchanged. Suppose further that the vaccine loses effectiveness against the new variant; it is now just 40% rather than 95%. 40% of 5% is 2%, the new monthly infection risk of those who have been vaccinated.
Effectiveness in the research sense has fallen from 95% to 40% from the first scenario to the second, smaller but still noticeably positive, but the risk of infection faced by someone who has been vaccinated in the second scenario is greater than the risk faced by someone unvaccinated in the first.
These numbers were made up, and of course the notion of a typical individual is a gross oversimplification, but the point applies to the current situation. We now have vaccines against the coronavirus, and we also have a new, much more transmissible variant. Effectiveness as researchers understand it has fallen, perhaps to about 40% for those vaccinated more than four months ago. It’s still very important to get vaccinated, since it reduces both the risk of infection and the risk of infecting others relative to not being vaccinated, but even so you may well be at greater risk of infection now than you were a year ago before the vaccines were introduced.
The bottom line: vaccine effectiveness measures the risk faced by vaccinated individuals compared to those who aren’t vaccinated. If the risk rises for the second group it rises for the first, even more if effectiveness is also falling.
Peter,
Thank you for starting this conversation. While the politics of the pandemic have replaced the medical, it is still worth figuring out the realities of ‘the science’ as well as how the money trail of doing business has severely limited our discussion of what is effective and what is safe. While phase three results for other vaccines are published, the US has had little public discussion about alternative vaccines effectiveness other than Moderna and Pfizer. Now that boosters are in the mix and federal monies announced and mandates being explored, media and ‘the science’ appear completely co-opted by current limited options.
Moderna says its COVID vaccine remains effective after six months
Boston Globe – August 5
Moderna’s vaccine for COVID-19 remained 93 percent effective six months after people received the second shot, the Cambridge company said Thursday, although the data was totaled before the emergence of the highly contagious Delta variant.
The results indicated that the messenger RNA vaccine’s efficacy barely declined from its peak of 94 percent, Moderna said in a statement before its second-quarter earnings call.
“We are pleased that our COVID-19 vaccine is showing durable efficacy of 93 percent through six months, but recognize that the Delta variant is a significant new threat so we must remain vigilant,” said Moderna’s chief executive, Stephane Bancel.
The data has yet to be published in a peer-reviewed medical journal and the company didn’t share other results, other than to say the data collection ended in March, before the Delta variant emerged.
The efficacy of a similar mRNA vaccine developed by Pfizer and its German partner BioNTech fell from 96 percent to 84 percent over six months, according to data released last Wednesday, a decline that could bolster Pfizer’s case that a third dose will eventually be needed. That data collection also ended before the emergence of Delta.
All three of Moderna’s booster-shot candidates produced “robust antibody responses” against Delta and other variants of concern in a mid-stage clinical trial, Moderna said. The boosters are being tested at a 50-microgram dose, half of what is used in the current shot. That data has been submitted to a journal for publication. …
Moderna says its Covid vaccine booster produced ‘robust’ immune response against delta
Moderna said its Covid-19 vaccine booster shot produced a “robust” antibody response against the highly contagious delta variant, according to details of a study released Thursday with the company’s second-quarter earnings report.
In a phase two trial, Moderna is testing a 50-microgram dose of three vaccine booster candidates in previously vaccinated individuals. The booster shots produced a promising immune response against three variants, including delta, with antibody levels approaching those seen in previously unvaccinated people who received two 100-microgram doses, the company said. …
Vaccine effectiveness is a ratio, the fraction of the control group that gets infected divided by the corresponding fraction of the treatment group; it’s a ratio of two ratios.
___________________________________________________
Not according to Pfizer. What Pfizer measured and compared was the amount of serious illness and death in the two groups. They made no effort to determine how many people in each group gactually ot covid. If it was not serious illness or death they weren’t interested. There was no attempt made collect data on how many were infected or how many tested positive.
This question is on the minds of those who are vaccinated and those contemplating vaccination, as well as physicians, vaccine developers, the CDC and government officials. Dorman writes: “ The everyday use of “effectiveness” is effectiveness against the virus. The research use is effectiveness relative to the control group. This is immense, but widely misunderstood and seldom explained.”
Yes, effectiveness in drug, device and vaccine trials is measured by comparing control/placebo group results to treatment group results. In the particular case of the COVID-19 vaccines, the trials are not challenge trials. It would be unethical to infect subjects with a potentially lethal virus like SARS-CoV-2. So the idea was that both groups would have some significant likelihood of natural infection that would be reportable in the trial (Disclosure: I’m in the Moderna Phase III trial).
Of course, we know that measured by the standards of their trials, the Moderna, Pfizer, J&J and AstraZenica vaccines all were “effective.” Since then, we have data from millions of vaccinated and unvaccinated people to back up that conclusion and it appears that the Pfizer vaccine will receive full FDA approval in September.
With the inevitable appearance of viral variants, the pressing question is how “effective” are the vaccines against each variant. The frontline assay has been to test antisera from vaccinated people for its ability to block virus from entering cells in culture. By that assay, all the vaccines have been deemed “effective,” albeit with somewhat lower effectiveness than for the original virus in some cases.
As Dorman acknowledges, one assay for vaccine “effectiveness” in the wild is whether it protects from infection. The problem with this is that most COVID-19 infections in both vaccinated and unvaccinated folks are inapparent, and those people are seldom tested, so we really don’t know what either the numerator or denominator are in those effectiveness calculations. More reliable are the statistics on hospitalization and deaths, and there, the performance of vaccines so far have been impressive.
Here’s an article from Nature that summarizes the current state of play. While the dominant theme is uncertainty, the data support the view that the vaccines are helping and that, until new vaccines are available, the best thing to fight new variants is for everyone to be vaccinated.
https://www.nature.com/articles/d41586-021-02054-z?WT.ec_id=NATURE-20210805&utm_source=nature_etoc&utm_medium=email&utm_campaign=20210805&sap-outbound-id=F8C7DC5800EB4926D8285EFA8E73396624E272C1
one assay for vaccine “effectiveness” in the wild is whether it protects from infection. The problem with this is that most COVID-19 infections in both vaccinated and unvaccinated folks are inapparent, and those people are seldom tested, so we really don’t know what either the numerator or denominator are in those effectiveness calculations.
______________________________________________
The vaccine manufacturers had a pretty good idea from the start that the vaccines would not be very effective at preventing infection. And more important the vaccines can not prevent contagious infection. It is becoming increasingly clear that the vaccinated are important vector for the spread of covid19.
The vaccine makers and the CDC have made things worse by not adequately monitoring the vaccinated for infection and by telling the vaccinated that they no longer need to worry about taking measures like wearing masks and physical distancing to prevent the spread .
So what could go wrong? Here is one doctors opinion on what might go wrong:
https://onlinelibrary.wiley.com/doi/full/10.1111/ijcp.13795
Jim:
You take issue with the “informed – consent” documentation?
The doctor that wrote the article who is an expert in the field says that people who are taking the vaccine are “not” being adequately informed to give proper consent.
The risk that people are not being told about is that at some point in time in the future the vaccines may backfire and actually cause more severe disease instead of less severe disease.
The article goes on to explain the mechanism by which the vaccines “may worsen COVID-19 disease”. There is no evidence that this is happening now, but there is evidence it could happen in the future and therefore individuals should be informed of this possible future risk.
The problem, as I see it, is people are being mislead into believing that the vaccines prevent infection. The reality is that the virus continues to circulate even among the vaccinated and the more the virus replicates the greater the chance of mutations. There is a possibility that a mutation could actually benefit from the antibodies created by current vaccinations. This would make that variant more dangerous to the vaccinated. For example, vaccines have been developed for SAR1 but never licensed in part because the risks of antibody-dependent enhancement.
@run – Thanks for correcting my mistake.
Maybe I’m just too in the weeds, but I think the medical definition is pretty straightforward. I doubt that the typical American has a “everyday” definition in their head as to what vaccine effectiveness would mean or that peoples’ “everyday” definitions would match.
I find the easiest way to explain it is with a hypothetical. Suppose someone is in a situation where they would be infected if unvaccinated. A 95% efficacious vaccine would prevent that infection 95% of the time.
If the endpoint is symptoms, hospitalization or death, just alter the hypothetical accordingly. For example:
Suppose someone is in a situation where they would be infected and go onto die if unvaccinated. A 95% efficacious against death vaccine would prevent that death 95% of the time.
Maybe the simplest takeaway from my OP is that, unless vaccine effectiveness increases for some reason (not an issue now), when the risk goes up for the unvaccinated it goes up correspondingly for the vaccinated. Public health messaging that conflates the risk faced by vaccinated people relative to the unvaccinated and the risk of a bad Covid outcome itself (absolute risk if you will) is unhelpful. Virtually every emanation from the CDC and the media parroting them does this.
The point about different outcomes is well taken. Risk of death and hospitalization are much easier to measure, especially since the CDC chose to stop collecting breakthrough case data except for cases involving death or hospitalization. There are other sources, however, and the recent Israeli study has stirred up a lot of interest. Speaking personally, I am annoyed at people who downplay infections that don’t qualify as “severe” in CDC terms. I’m acutely vulnerable to kidney impairment. I have close relatives with other vulnerabilities. It would be bad news for any of us to get Covid, even if the initial symptoms are “mild”.
In any case, in my OP I used infection as the sole metric for convenience. The argument applies equally to any other metric, of course.
What are the estimates of post-infection natural immunity efficacy? Someone I know well was advised in April that her status as confirmed COVID of early January was at least on par with vaccination and to consider vaccination in about October. Oddly, last month this was changed to “get it now” but her doctor confided that the initial advice was still medically sound. Not confidence instilling.
@Eric,
The challenge to answering that question is that the dose of virus in a natural infection can be highly variable, whereas the dose of antigen in the vaccines is more controlled. Probably the best way to answer this would be to do spike antibody testing post-infection and compare to the average level of antibody in the vaccinated population. If I had a natural infection and my spike antibody levels compared favorably to the typical vaccinated person, I’d feel pretty good.
Note that some hospitals test for the nucleocapsid antibody, not the spike protein. Antibody to the viral nucleocapsid is a good marker for prior infection, but is not protective.
how long effectiveness lasts should also be addressed….Pfizer has been pushing for a third shot for a while now, and Israel has started administering them; Moderna now says a booster shot will be needed this fall…
WHO, on the other hand, says we should get the poor countries their first shot before worrying about third shots for the rich countries…