Assessing consistency in meta-analysis: a new measure considers statistical power

Researchers have improved the assessment of consistency in meta-analysis. The improved consistency measure considers statistical power, and it has potential to alter the interpretation of meta-analyses. The new measure was published in European Journal for Philosophy of Science.

Meta-analysis refers to combining results of several studies mathematically. Especially, medicine considers meta-analyses as the highest level of evidence. If a meta-analysis recommends a treatment, the treatment most probably becomes established.

Meta-analysis must be consistent per se. Conventional measures of consistency straightforwardly favour large studies and do not consider statistical power as a source of inconsistency. Statistical power denotes the ability of a study to detect statistically significant results, such as effects of a treatment.

“Repeating a study with high statistical power is more difficult than repeating a study with low statistical power. Our proposed measure of inconsistency emphasises this aspect,” says Data Manager Ari Voutilainen from the University of Eastern Finland, one of the four developers of the measure.

“If a meta-analysis combines studies that differ from each other with respect to statistical power, the meta-analysis is inconsistent, even if within-study variances are low. Moreover, it is important that studies with the highest statistical power have the strongest effect on the results of a meta-analysis. Our proposed measure accentuates the meaning of statistical power also from this viewpoint.”

Scientific journals publish approximately half a million meta-analyses annually. Consequently, the improved consistency measure can have a substantial effect on research in general.

For further information, please contact:

Data Manager Ari Voutilainen, ari.voutilainen(at)uef.fi, tel. +358 40 355 2826, ari.voutilainen@uef.fi
https://uefconnect.uef.fi/en/person/ari.voutilainen/

Research article:

Berchialla, P., Chiffi, D., Valente, G. & Voutilainen, A. The power of meta-analysis: a challenge for evidence-based medicine. Euro Jnl Phil Sci 11, 7 (2021). https://doi.org/10.1007/s13194-020-00321-w