Is Retrospective Study Reliable in Genetic Studies?
To the Editor:
We read with great interest the preliminary study by Niskakangas et al,1 “Association of Apolipoprotein E Polymorphism With Outcome After Aneurysmal Subarachnoid Hemorrhage.” Although in this paper the authors concluded that there is a significant association between apolipoprotein E ε4 (apoE ε4) and the outcome of subarachnoid hemorrhage (SAH) patients, we have some reservations about the methodology and its data analysis.
The classification of CT findings used in this study was different from the original study of Fisher et al2: there are only 4 grades in Fisher’s grading scale, instead of the 5 grades shown in the article by Niskakangas et al. Each grade was clearly defined in the Fisher study, whereas the scale used in the Niskakangas study falls short of a clear definition. In their statistical analysis section, the authors then mention that the CT findings were allocated to 1 of the 4 categories. It would be helpful if the authors could clarify how they grouped the 5 grades into 4 categories during the statistical analysis.
We noticed there was a fundamental problem with the data collection and presentation on the neurological outcome of patients. Patients from groups both with and without apoE ε4 had a 0- to 57-month follow-up. This may contribute to inaccuracy and bias in the assessment of the patients neurological outcome after a SAH. The convention in neurological outcome assessment prescribes a duration of at least 6 months from the time of hemorrhage to the time of assessment, in order to ensure that the patient’s clinical condition and recovery are stable before assessment.3 Information on the distribution (SD) of the duration of follow-up may indicates the proportion of patients who had their neurological outcome assessment too early during their recovery period. The other possibility is that these 0-month follow-up cases were all mortalities.
Results presented in Tables 1, 2, and 3 of the article by Niskikangas et al1 may contain an error. All variables, including the demographic, clinical, and radiological data and the neurological outcome, that compared with the apoE ε4 group were not carried out by the Fisher exact test as mentioned in the Statistical Analysis section. The probability values presented were, in fact, the standard errors of the difference of the 2 proportions compared; neither the scale of measurement was comparable (eg, 0.029 instead of 2.9%) nor was the calculation was based on the exact method.
Apart from the misleading probability value calculated, it was inappropriate to compare each subgroup separately. For example, the authors compared each level of initial Hunt and Hess score with and without apoE ε4 separately using the same data. Instead, the Fisher exact test (or χ2 test), as originally proposed, for this 5×2 contingency table is the appropriate method.
The authors mentioned that there were 4 patients who were dead, but their apoE genotypes were unknown. We have also included these patients into the analysis, assuming that none, 1, 2, 3, or all 4 harbor the apoE ε4 allele. The association between apoE ε4 and outcome disappeared when none of these patients were assumed to have the gene (P=0.068; OR 2.15, 95% CI 0.94 to 4.97).
This article clearly demonstrated the limitation of a retrospective study. It is difficult to convince the readers that there is an association between APOE polymorphism and SAH recovery with a retrospective study as shown in this paper, especially when there was a significant number of missing patients (14.3%). Although we have queries regarding the methodology, data analysis, and the conclusion of this article, it did demonstrate the necessity of having a well-designed prospective clinical study that may be in a better position to investigate the association between apolipoprotein E polymorphism and outcome in patients with aneurysmal SAH.
We wish to thank Poon et al for their interest and very valuable comments concerning our recent article in Stroke.1 We agree that we did not use the original Fisher’s grading scale, but properly noted that a modified Fisher scale was used. We should have defined the grading system more clearly already in the article. Differences are as follows: in grade 1, we saw only a very small amount of blood locally in the subarachnoid space (can be seen); in grade 2, there was subarachnoid blood <1 mm thick; grade 3, >1 mm thick, diffusely spread throughout the subarachnoid cisterns; and in grade 4, there was diffuse subarachnoid hemorrhage (SAH) and/or large hematoma or intraventricular bleeding. Although we did not use the original Fisher grading scale, in our opinion it was mainly based on that, and therefore we considered it fair to reference the original article. Regrettably, there is an error in the text in the Statistical Analysis section. We have allocated CT findings to 5 groups in statistical analysis, 0 through 4.
We agree that there is heterogeneity in the follow-up times of the patients. However, the length of the follow-up time partly describes the outcome of a patient, and the different distribution of the ε4 allele is already seen here. Patients with good outcome returned to normal life after a convalescence period of approximately 2 to 3 months, were clinically stable, and had no clinical indication to be followed up after that. Patients who were primarily in poor condition were followed up for a longer period, enabling us to better estimate the final clinical condition of these patients. The majority of those 0-month follow-up cases were patients who died, and in addition 3 patients (followed up for 1 to 2 weeks) who were primarily in good condition when leaving the hospital.
As Poon et al point out, there were unforgivable calculation errors concerning the statistics of the patient characteristics (Table 2). The probability values in the Table 2 are erroneous statistical raw data. The differences between the patients with and those without the APOE ε4 allele were tested using the χ2 test. All probability values for the group differences were statistically insignificant, as follows. In the Table 2, concerning the initial Hunt and Hess score, the P value was 0.21 (df=4, χ2=5.82), and in the initial CT findings the P value was 0.91 (df=4, χ2=0.97).
Poon et al have presented an interesting view about considering those 4 missing dead patients, presenting 3% of the whole series. That was very valuable point, and it emphasizes the problem of the selection bias in retrospective association studies. We hypothesized that the ε4 allele is a risk factor for poor outcome after SAH. In a recent publication by Dunn et al,2 the ε4 allele frequency of SAH patients was lower than in normal population, suggesting that the ε4 allele could be associated with an excess rate of early mortality after SAH and therefore causing a selection bias, while patients with poor outcome are not included to clinical patient series. The same could have happened in our series.
Poon et al have stated some of the potential weaknesses of retrospective study design in genetic studies in general and in our study in particular. The complexity of the natural cause of this devastating disease and possible complications in therapy itself make the design of outcome studies quite difficult. Thus far, there has been one prospective clinical series studying the association of APOE genotype with SAH outcome, and even in it there were some problem with selection bias.
We do not suggest that our hypothesis is the final truth. It is a preliminary report, as we stated in the subtitle of our article. There is, indeed, a need for a prospective, properly designed population-based studies that includes patients who die before admission to hospital. Future studies will show whether the APOE gene polymorphism has a role in the prognosis of subarachnoid hemorrhage. Considering the fundamental role of APOE in neurobiology,3 we think that APOE is an important area for studies of subarachnoid hemorrhage and of hemorrhagic stroke in general.