Pitfalls in Meta-Analysis of Observational Studies: Lessons From a Systematic Review of the Risks of Stenting for Intracranial Atherosclerosis
To the Editor:
Stenting is a potentially efficient strategy in addition to medical treatment in patients with symptomatic intracranial stenosis, but to date, there are no data from randomized controlled trials showing its efficacy. Hence, the risk–benefit balance of this technique can only be assessed by comparing absolute risks of events estimated in observational studies. Groschel et al performed a useful systematic review of the risks of intracranial arteries stenting, but we have concerns about the methods used to summarize the results.1
First, Groschel et al did not formally assess heterogeneity (ie, variability in estimated risks) across studies. Heterogeneity, which should be measured and addressed in any meta-analysis, can be measured using a χ2 test and the I2 statistic, the latter being the percentage of the variability in effect estimates that is due to heterogeneity rather than sampling error (chance). In case of substantial heterogeneity, one analytic approach is to incorporate it into a random-effects model.2
Second, Groschel et al estimated the average risk of stenting by calculating the median of individual risks (with 25% and 75% percentiles), but they did not use any weighting. This approach can lead to a biased estimation of the average risk, because small and large studies contribute equally. Yet, it is well established that in meta-analyses, more weight should be given to studies with a large amount of information. Meta-analysis of proportions may be treacherous and several methods are available to calculate pooled estimates of proportions. A common method is to compute a weighted average of the individual proportions estimated from each study, in which the weights are study sample sizes. This method can be enhanced with the use of a correction for the CI of the pooled proportion to deal with overdispersion,3 which refers to the fact that the variability in observed data can be greater than that expected under the theoretical model. Overdispersion is particularly common when proportions are <0.30 or >0.70.4 However, this simple method of pooling with or without correction for overdispersion is valid in the absence of heterogeneity only. The use of the Freeman-Tukey transformation seems to be a better alternative.5 This is a variance-stabilizing transformation that removes the dependence of the variance on the mean of the transformed proportion (ie, it corrects for overdispersion). In addition, individual transformed proportions can be combined with standard meta-analytic methods, including a random-effects model to incorporate heterogeneity, the pooled estimate being backtransformed afterward (detailed method is available on request from the authors). This method has been used in recent meta-analyses.6,7
Third, meta-analysis of comparative risk estimates requires appropriate methods, which have been extensively described elsewhere.8 To compare risks of stenting in anterior and posterior circulation patients, Groschel et al erroneously derived an OR from the total numbers of events and that of treated patients in each group across studies without any weighting and by incorporating studies that enrolled patients with anterior or posterior circulation intracranial artery stenosis only.
From the data set provided in the Groschel et al article, we found that there was substantial heterogeneity in individual estimates of the periprocedural risk of stroke or death across studies (χ2 test probability value=0.02; I2=47%; 95% CI, 12 to 65). When using the Freeman-Tukey transformation and a random-effects model, we estimated that the combined periprocedural risk of stroke or death was 10.5% (95% CI, 8.0 to 13.4). Using the Mantel-Haenszel fixed-effect method and including studies only in which both anterior and posterior circulation patients were enrolled, we found that the risk of stroke or death after intracranial artery stenting in the posterior was not significantly higher than that in the anterior circulation (OR, 1.16; 95% CI, 0.65 to 2.08; heterogeneity χ2 test probability value=0.72; I2=0%, 95% CI, 0 to 51; Figure).
Our conclusion is therefore noticeably different from that of Groschel et al, because we found that the overall risk of stroke or death after intracranial artery stenting is, with 95% confidence, at least equal to 8.0% and as high as 13.4%. Moreover, our results do not support that risks differ between anterior and posterior circulations.
Gröschel K, Schnaudigel S, Pilgram S, Wasser K, Kastrup A. A systematic review on outcome after stenting for intracranial atherosclerosis. Stroke. 2009; 40: e340–e347.
McCullagh P, Nelder J. Generalized Linear Models. Boca Raton: Chapman & Hall; 1989.
Zar J. Biostatistical Analysis. Englewood Cliffs, NJ: Prentice Hall; 1998.
Stasi R, Sarpatwari A, Segal JB, Osborn J, Evangelista ML, Cooper N, Provan D, Newland A, Amadori S, Bussel JB. Effects of eradication of Helicobacter pylori infection in patients with immune thrombocytopenic purpura: a systematic review. Blood. 2009; 113: 1231–1240.
Deeks J, Altman D, Bradburn M. Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. In: Egger M, Smith G, Altman D, eds. Systematic Reviews in Health Care: Meta-Analysis in Context. London: BMJ Publishing Group; 2001.