Risk of Bias in Test-Negative Studies on COVID-19, Influenza Vaccine Effectiveness

The risk of confounding bias in regard to estimated vaccine effectiveness (VE) was low for COVID-19 and influenza test-negative studies and relied on correlated vaccine behaviors, according to results of a simulation study published in Clinical Infectious Diseases.

To evaluate the risk of confounding bias in VE from test-negative studies, vaccination rates for COVID-19 and influenza were simulated in populations of individuals under differing parameters. The probability of being vaccinated correlated with factors such as motivation, age, and vaccine efficacy. Mean bias was defined as the difference between estimated and true VE, and Absolute mean bias of estimated VE was calculated as either low (<10%), intermediate (10%-<20%), or high (³20%). In all simulations, the researchers assumed a 55% and 70% rate of vaccination against influenza and COVID-19, respectively, which was based on vaccine uptake among US adults between 2020 and 2021.

For COVID-19 VE, all simulations that included a control group with influenza underestimated the true VE. In simulations in which a control group with influenza represented 25% or less of the population, the researchers observed that mean bias was decreased. However, simulations in which the number of control patients with influenza approached 50% of the study population, moderate to high bias was observed in regard to estimated COVID-19 VE.

Similar patterns were observed in all simulations that assessed bias in estimated influenza VE. The true influenza VE was underestimated in all simulations that included a COVID-19 control group. Moderate to high bias was observed in most simulations in which the COVID-19 control group represented at least 75% of the study population.

This study may be limited as other COVID-19 test negative VE studies may include subgroups of patients — particularly older adults and those at increased risk for severe disease — for whom the conditional probability of vaccination is increased.

According to the researchers, “our work suggests low bias in VE estimates derived from COVID-19 and influenza test-negative studies with influenza and SARS-CoV-2 controls, respectively, in situations where these controls represent a low proportion of total test-negative controls.” In regard to test-negative studies in which the risk of bias is not meaningful, the researchers concluded that “adequate justification should be provided to promote critical interpretation and confidence in study results.”

Reference

Doll MK, Pettigrew SM, Ma J, Verma A. Effects of confounding bias in COVID-19 and influenza vaccine effectiveness test-negative designs due to correlated influenza and COVID-19 vaccination behaviors. Clin Infect Dis. 2022;ciac234. doi:10.1093/cid/ciac234