External Validation of the Secondary Intracerebral Hemorrhage Score in The Netherlands
Background and Purpose—We aimed to validate externally in a setting outside the United States the secondary intracerebral hemorrhage (ICH) score that was developed to predict the probability of macrovascular causes in patients with nontraumatic ICH.
Methods—Patients with nontraumatic ICH admitted to the University Medical Center Utrecht, the Netherlands, between 2003 and 2011 were included if an angiographic examination, neurosurgical inspection, or pathological examination had been performed. Secondary ICH score performance was assessed by calibration (agreement between predicted and observed outcomes) and discrimination (separation of those with and without macrovascular cause).
Results—Forty-eight of 204 patients (23.5%) had a macrovascular cause. The secondary ICH score showed modest calibration (P=0.06) and modest discriminative ability (c-statistic 0.73; 95% confidence interval, 0.65–0.80). Discrimination improved slightly using only noncontrast computed tomography categorization (c-statistic 0.79; 95% confidence interval, 0.72–0.86).
Conclusions—The discriminative ability and calibration of the secondary ICH score are moderate in a university hospital setting outside the United States. Clues on noncontrast computed tomography are the strongest predictor of a macrovascular cause in patients with ICH.
Nontraumatic intracerebral hemorrhage (ICH) is the subtype of stroke with the highest case fatality.1 Approximately 15% to 20% of patients with nontraumatic ICH have an underlying macrovascular abnormality cause, such as an arteriovenous malformation, a dural arteriovenous fistula, or an aneurysm. Early identification of these lesions has important therapeutic and prognostic consequences.2,3
Computed tomography (CT) angiography enables assessment of vascular pathology in the emergency setting, but it is unclear which patients with ICH should undergo angiographic examinations.4 The secondary ICH (SICH) score was developed and validated in the United States to identify patients with a high risk of an underlying macrovascular cause on the basis of patient and hemorrhage characteristics (Table I in the online-only Data Supplement).5 The same investigators performed an external validation study in another US hospital, which indicated that the SICH score is an accurate tool in US settings.6
The discriminative power of the SICH score, however, may depend on population differences and local variation in which patients with ICH undergo angiographic studies to find a macrovascular cause. We, therefore, sought to validate externally the SICH score in a setting outside the United States and to assess the effect of the separate items of the SICH score on the probability of harboring an underlying vascular cause.
Materials and Methods
From a prospectively collected database of patients with stroke admitted to the University Medical Center Utrecht, the Netherlands, we included patients admitted between February 2003 and May 2011 who met the following inclusion criteria: (1) nontraumatic ICH on noncontrast CT (NCCT); (2) ≥18 years; and (3) ≥1 angiographic study (CT angiography, MR angiography, or digital subtraction angiography), or pathological examination.
Age, sex, and history of hypertension or impaired coagulation were retrieved.5 NCCTs were reviewed independently by 2 observers blinded to clinical data and final diagnosis. Each NCCT was assessed by a neurology (C.J.J.v.A.) or radiology (P.J.v.L.) resident, and a neurology (C.J.M.K.) or radiology (B.K.V.) staff member. NCCTs were classified as high probability of finding an underlying macrovascular cause when enlarged vessels or calcifications were present along the ICH margins, or visible hyperattenuation within a dural venous sinus or cortical vein. In low-probability NCCTs, none of these were present and the hematoma was located in basal ganglia or brain stem. Indeterminate probability NCCTs fulfilled neither high- nor low-probability criteria.5 Differences in reader interpretations were resolved in consensus meetings.
The outcome measure was a macrovascular cause of the ICH identified by CT angiography, MR angiography, or digital subtraction angiography, neurosurgical inspection, or pathological examination.
Interobserver agreement for NCCT categorization was determined with κ statistics. To assess the effect of separate items of the SICH score on the probability of an underlying vascular cause, we performed univariable logistic regression analyses for all predictors separately in both the Boston derivation cohort and Utrecht validation cohort. Subsequently, we calculated the SICH score and predicted probability for each patient in the validation cohort on the basis of original regression coefficients of the final multivariable model on which the SICH score was based.5
We assessed performance of the SICH score by discrimination (separation of those with and without vascular cause) and calibration (agreement between observed and predicted outcomes). Discrimination was assessed using the area under the receiver-operating curve. For calibration, we constructed a calibration curve plotting the observed versus the predicted probability and performed the Hosmer–Lemeshow test.
From 517 patients admitted with nontraumatic ICH, we included 204 patients. Baseline characteristics of the Boston derivation and the Utrecht validation cohorts are listed in Table II in the online-only Data Supplement. In our validation cohort, an underlying vascular cause was identified in 48 of 204 patients (23.5%), in comparison with 91 of 623 patients (14.6%) in the derivation cohort (Table III in the online-only Data Supplement). In the derivation cohort 273 patients (43.8%) and in our validation cohort 46 patients (22.5%) were aged 71 to 94 years. The female preponderance in patients with a macrovascular cause in the derivation cohort was not observed in our validation cohort. Validation cohort patients less often had hypertension (59.1% versus 42.2%) or impaired coagulation (32.7% versus 21.6%) than derivation cohort patients.
Interobserver agreement for NCCT categorization in our validation cohort was substantial (κ statistic 0.64; 95% confidence interval [CI], 0.55–0.73). In our validation cohort, 66 NCCTs (32%) were categorized as low probability, 116 (57%) as indeterminate, and 22 (11%) as high probability.
In the univariable analyses, the effect of the predictors was of comparable magnitude in the derivation and the validation cohort with the except for sex (odds ratio derivation cohort, 1.6; odds ratio validation cohort, 0.7; Table II in the online-only Data Supplement). The strongest predictor of harboring an underlying macrovascular cause was NCCT categorization.
In the validation cohort, the SICH score showed modest discriminative ability (c-statistic 0.73; 95% CI, 0.65–0.80), with a point estimate outside the 95% CI of the point estimate of the c-statistic in the derivation cohort (0.87; 95% CI, 0.84–0.89; Table). Discrimination was much better when applying the original regression coefficients of the final multivariable model of the derivation group (c-statistic 0.81; 95% CI, 0.75–0.88). The mean predicted probability of finding a macrovascular lesion (22.5%) was comparable with the mean observed presence of a macrovascular lesion (23.5%). Calibration was only modest (Hosmer–Lemeshow test, P=0.06; Figure I in the online-only Data Supplement). Discrimination was slightly better using only NCCT categorization (c-statistic 0.79; 95% CI, 0.72–0.86).
The SICH score has only moderate discriminative ability in a Dutch university hospital. NCCT classification, based on the presence of ICH characteristics suggestive of an underlying macrovascular cause and location of the ICH, was the strongest predictor of a macrovascular cause.
Our finding that the discriminative ability of the SICH score was lower in a Western European setting than in the United States is probably caused by differences in patient characteristics between the derivation and the validation cohort. The patients in our cohort were younger and less often had impaired coagulation or hypertension. The yield of angiographic examinations is known to be higher in normotensive patients and those aged <55 years.4,7 Indeed, the yield of angiographic examinations was higher in our cohort than in the derivation cohort.
A strength of our study is that we were able to study the external validity of the SICH score in a large cohort of patients with ICH on a different continent, with a different approach to performance of angiographic studies in patients with ICH. In addition, our study shows that NCCT characteristics suggestive of an underlying vascular cause may not always be easy to recognize and requires training.
A limitation of our study is that not all patients underwent digital subtraction angiography and small arteriovenous malformations and dural arteriovenous fistulas may have been missed.
In the Dutch setting, NCCT categorization should be used to determine whether there is a high likelihood of finding an underlying macrovascular cause in patients with ICH. Further studies are needed to determine whether performance of angiographic studies in patients with low-probability NCCT is cost-effective and to determine which type of angiographic study should preferably be applied.
Dr Almandoz kindly provided the absolute numbers of patients per secondary intracerebral hemorrhage score category and results of the multivariable logistic regression model of the derivation cohort.
The study was supported by the Netherlands Heart Foundation (grant no. 2007B048 to Dr Klijn). The other authors report no conflicts.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.113.002386/-/DC1.
- Received June 5, 2013.
- Accepted June 7, 2013.
- © 2013 American Heart Association, Inc.
- Ohkuma H,
- Tsurutani H,
- Suzuki S
- Cordonnier C,
- Klijn CJ,
- van Beijnum J,
- Al-Shahi Salman R
- Delgado Almandoz JE,
- Schaefer PW,
- Goldstein JN,
- Rosand J,
- Lev MH,
- González RG,
- et al
- Zhu XL,
- Chan MS,
- Poon WS