Prediction of Major Vascular Events in Patients With Transient Ischemic Attack or Ischemic Stroke
A Comparison of 7 Models
Background and Purpose—In patients with a recent TIA or minor stroke, prediction of long-term risk of major vascular events is important, but difficult. We aimed to study the external validity of currently available prediction models.
Methods—We validated predictions from 3 population-based models (Framingham, SCORE, and INDIANA project) and 4 stroke cohort-based models (Stroke Prognosis Instrument II, Oxford TIA, Dutch TIA study, and the ABCD2 study) in an independent cohort of patients with a recent TIA or minor stroke. The validation cohort consisted of 592 patients with TIA or minor stroke, with a mean follow-up of 2 years. The primary outcome was the 2-year risk of the composite outcome event of nonfatal stroke, myocardial infarction, or vascular death. We used calibration graphs and c-statistics to evaluate the 7 models.
Results—The 2-year risk of the primary outcome event was 12%. Calibration was adequate for stroke population-based studies. After adjustment for baseline risk and for prevalence of risk factors, calibration was adequate for the Dutch TIA, the ABCD2, and Stroke Prognosis Instrument II models. Discrimination ranged from 0.61 to 0.68.
Conclusions—Discrimination was poor for all currently available risk prediction models for patients with a recent TIA or minor stroke, indicating the need for stronger predictors. Clinical usefulness may be best for the ABCD2 model, which had a limited number of easily obtainable variables, a reasonable c-statistic (0.64), and good calibration.
Patients with a recent stroke have an increased long-term risk of new cerebrovascular and cardiovascular events. Estimates of long-term risk of cardiovascular and cerebrovascular events are important for patients who want to know their individual risks, and for treating physicians because in patients with increased long-term risk, more expensive or more hazardous interventions could be worthwhile. Long-term risk seems to be dependent on the underlying risk factors.1
In an American Heart Association statement, it was pointed out that validated prediction models for long-term cardiovascular risk in patients with ischemic stroke or TIA were not available.2 The use of a population-based prediction model, ie, the Framingham risk score,3 was recommended; however, this model has not been evaluated and compared with other prediction models to predict long-term cerebrovascular and cardiovascular risk in clinical practice. Treatment recommendations based on predictive models have been published and are currently used for patients with a significant carotid artery stenosis and for patients with atrial fibrillation.4,5 For patients without atrial fibrillation or a carotid artery stenosis, more precise risk estimations are needed.6,7 For this purpose, we need a reliable prognostic model with good discrimination between patients who will have new events and those who will not.
The aim of the present study is to determine which prediction models are adequate for long-term risk estimation for cardiovascular or cerebrovascular events in TIA and minor stroke patients. We primarily focused on major vascular events, ie, nonfatal stroke, myocardial infarction, and vascular death, because these affect health status most and are the focus of intervention studies. Secondarily, we considered fatal and nonfatal stroke.
Materials and Methods
Selection of Prediction Models
We compared the external validity (ie, calibration and discrimination) of 3 population-based models (the Framingham risk score for cardiovascular events,3 the Framingham risk score for stroke,8 SCORE,9 and the INDIANA project cardiovascular risk score10) and 4 stroke cohort-based models (Stroke Prognosis Instrument [SPI] II,11 the Oxford TIA,7 the Dutch TIA study,12 and the ABCD2 score13) in an independent cohort of patients with a recent TIA or minor stroke (Table 1). All models were designed to predict long-term outcomes, except for the ABCD2 score, which predicts 90-day risk of stroke. The stroke population-based models were designed to predict events after TIA or stroke; the population-based models were designed to predict first-ever stroke or cardiovascular events. These models were based on well-described large cohorts of patients with TIA or minor stroke and had the relative risks of the predictors estimated in a multiple (logistic or proportional hazards) regression model. The models predicted stroke and other cardiovascular events.
The SPI-I was developed in a small cohort of patients with a recent stroke. Kernan et al11 created SPI-II by incorporating new predictive variables identified in a cohort of patients from the WEST study and validated it in 3 cohorts of patients from several secondary prevention trials.
Hankey et al7 developed a prediction equation for recurrent vascular events based on 8 clinical prognostic factors from a cohort of 469 hospital-referred TIA patients. The prediction model was validated in cohorts of TIA patients from the Oxfordshire Community Stroke Project and the UK-TIA study.14
The Dutch TIA trial investigators developed a set of predictors of major vascular events and of stroke. Their prediction model was based on data of patients with a TIA or minor stroke who entered a multicenter, randomized, controlled clinical trial with a 2×2 factorial design of high-dose vs low-dose aspirin and propanolol vs placebo.12,15
The ABCD2 score calculates the 2-, 7-, and 90-day stroke risk in TIA patients.13 This prediction model was based on TIA patients presenting within 1 day of onset of symptoms. We used the 90-day risk score because this was closest to the 2-year risk calculated in the validation cohort.
The 3 other models were population-based models that have been widely used to assess cardiovascular risk. For risk of major vascular events, we used the Framingham risk score for cardiovascular events published in 1998.3 This model is based on categorical variables. For risk of stroke, we used the Framingham risk score for stroke published in 1994.8
The SCORE prediction model, methodology differs slightly from the other models. Annual age- and gender-dependent rates of coronary and vascular event were estimated from the data in an accelerated failure time model (Weibull). This model made use of age as a measure of exposure time to risk rather than a risk factor. Estimates of mortality rates were based on observations in age categories.9
Pocock et al10 published a prediction instrument based on data from 8 randomized controlled trials of antihypertensive treatment in asymptomatic individuals: the INDIANA project. They developed 3 separate models for overall cardiovascular mortality, fatal coronary heart disease, and fatal stroke.
The validation cohort consisted of 592 consecutive patients included in the Rotterdam Transcranial Doppler study.16 Patients were recruited in our hospital. This study was designed to evaluate the diagnostic and prognostic value of transcranial Doppler ultrasonography in patients with a recent TIA or minor ischemic stroke. Patients were 18 years and older and had the index event within the preceding 6 months. They were independent (a score of ≤3 on the modified Rankin scale),17 and the TIA or stroke was of presumed atherosclerotic origin; this implied that patients with a mechanical heart valve and a proven dissection were excluded. Patients with atrial fibrillation or a significant symptomatic carotid artery stenosis were also excluded. Follow-up started at time of recruitment. The mean time between index event and recruitment was 51±45 days. The mean follow-up in this study was 755 days or 2.1 years (interquartile range, 0.9–3.1). The study population consisted of patients for whom no other treatment than risk factor modification and antiplatelet medication were available.
Procedure and Definitions
The primary outcome was the composite end point of stroke, myocardial infarction, and vascular death. The secondary outcome was the composite end point of fatal or nonfatal stroke. In the validation cohort, stroke was defined as a focal neurological deficit, resulting in disability for >24 hours, confirmed by a neurologist, with CT scan, if possible.
Myocardial infarction was defined as an episode of precordial chest pain accompanied by ECG evidence of recent infarction or release of cardiac enzymes. Vascular death was death within 4 weeks after stroke or myocardial infarction, or sudden death. Follow-up was performed annually by telephone survey by an experienced nurse. If patients reported any hospital visits or any symptoms suggesting a vascular event, then their treating physicians were contacted for further information or discharge letters. In the validation cohort, Kaplan-Meier analysis of survival was used to estimate the 2-year risk of the primary and secondary outcome.
The validation procedure was similar for each prediction model. First, we estimated the actual 2-year risk of primary and secondary outcome events for each patient in the validation cohort using Kaplan-Meier survival analysis. For missing values, the mean value of a variable was imputed. We then calculated the 2-year risk as predicted by the seven models. To do so, we used the original prediction equations of each model. Each equation was adjusted for the mean overall risk and for the prevalence of risk factors in the validation cohort, as recommended by several authors.18,19
Calibration and Discrimination
Two aspects of validity were examined: calibration and discrimination. Calibration, or reliability, measures how closely predicted outcomes agree with actual outcomes. Discrimination refers to the ability to distinguish patients with different outcomes.20 The predicted probabilities of vascular events should be trustworthy (calibration) and extreme (discrimination).
Calibration-in-the-large reflects whether the overall outcome of the study cohort was close to the average predicted 2-year risk from a model. We constructed calibration plots in which observed 2-year outcome was plotted against predicted 2-year risk (using a Kaplan-Meier survival curve). We indicate outcome by quintiles of predicted probabilities. Calibration curves can be approximated by a regression line (or calibration line) with intercept (α) and slope (β). Well-calibrated models have α=0 and β=1. We calculated slopes for each prediction model. Discrimination was quantified with the concordance (c) statistic. The c-statistic resembles the area under the receiver-operating characteristic curve. A c-statistic of 1 implies a test with perfect sensitivity and specificity, whereas a value of 0.5 implies that the model predictions are no better than chance.20
Comparability of the Cohorts
We first compared the distribution of prognostic factors in the 7 derivation cohorts (Table 2). People in the population-based studies from the INDIANA project and Framingham were younger than patients in the stroke cohorts and, by definition, did not have a history of stroke or myocardial infarction. For the SCORE population, the mean age or age distribution was not stated. In this study, relatively young subjects had participated, and subjects aged 40 to 60 were particularly well-represented. In the Oxford TIA cohort, all patients had a TIA, and approximately one-third had amaurosis fugax (Table 2). In the validation cohort, 57 major vascular events occurred during follow-up, for a 2-year risk of 12%, and 41 fatal or nonfatal stroke, for a 2-year-risk of 9% (Table 3).
Table 4 shows the observed predicted 2-year risk of stroke, myocardial infarction, or vascular death. Table 5 shows the observed predicted 2-year risk of fatal or nonfatal stroke. The SPI-II overestimated the risk for both outcome events in patients at low risk and underestimated in patients at high risk, whereas Oxford TIA, the Dutch TIA trial, and ABCD2 score overestimated the risk for both outcome events (Figure⇓).
Discrimination varied between 0.61 and 0.68 for the risk of stroke, myocardial infarction, or vascular death, and between 0.58 and 0.65 for the risk of fatal or nonfatal stroke (Table 4). Discrimination was higher in the stroke cohort-based studies than in the population-based studies but remained <0.70.
This is the first study to our knowledge comparing the external validity of long-term prediction models. In a scientific statement for health care professionals from the Stroke Council and the Council on Clinical Cardiology of the American Heart Association/American Stroke Association, the Framingham study3 was recommended for long-term cardiovascular risk in patients with TIA or ischemic stroke.2 The purpose of our study was to find out whether stroke cohort-based studies perform better and should be used in the future to estimate long-term risk of major vascular events in patients with TIA or ischemic stroke.
Calibration was fairly good in most models; however, in the SPI-II study, the Dutch TIA study, and the ABCD2 study, calibration was best. All models were adjusted for baseline risk and prevalence of risk factors, as suggested by several authors.18,19 Without these adjustments, calibration was poor. Overall, stroke-based prediction models from the Dutch TIA trial, the SPI-II study, and the ABCD2 study showed best calibration and discrimination. A clinically useful prediction model shows good calibration, discrimination, and uses a limited number of clear-cut, easily available variables.
The ABCD2 score and the SCORE prediction model used the smallest number of variables in their prediction rule for long-term cardiovascular risk, 5 and 6, respectively, and risk estimation derived from these studies therefore is easy to obtain. The Framingham prediction model for cerebrovascular risk included 8 factors, which are all easy to obtain, except for left ventricular hypertrophy for which an ECG is needed. The Dutch TIA study used 17 variables in the prediction rule. The Oxford TIA prediction rule and the INDIANA project prediction rule included ECG variables, which are more difficult to ascertain. The Dutch TIA prediction rule used ECG variables that require some expertise, such as left ventricular hypertrophy by the Casale criteria21 and anteroseptal infarction (Table 1), and also included CT scan variables. This makes these models less attractive to use.
Limitations of the Study
Some limitations of this study should be discussed. First, although the validation cohort was of reasonable size (598 patients), the effective sample size was not large, with 57 major events and 41 fatal or nonfatal strokes. Second, predictions from all models, except for the ABCD2 study, exceeded the 2- year risk observed in our validation cohort. We have overcome this miscalibration by adjusting for this difference in follow-up in our analysis.
We excluded patients with atrial fibrillation or a severe carotid artery stenosis and focused on patients with TIA or minor ischemic stroke. This reduces the heterogeneity of the validation cohort and may reduce the discriminatory ability of the models. Risk estimates are more readily available for carotid artery stenosis, as well as for patients with atrial fibrillation. For the remaining group of patients with TIA or minor stroke, there is a particular need for a prediction model.
In the validation cohort as well as in 2 of the 4 stroke cohorts, the mean age was rather low (Table 2). These cohorts consisted of a hospital population. Hospital populations tend to be of lower age. Therefore, they may not be representative of patients with TIA and minor strokes in the community.
The time between index event and recruitment in the validation cohort was 51 days, which differs substantially from the ABCD2 study but not the remaining stroke-based population studies. We have overcome this problem by adjusting for the baseline risk in all validation studies. In this way, we adjusted for the low number of outcome events compared to the ABCD2 study.
Although this was a study on long-term risk, we also included the ABCD2 model, which was developed for estimation of short-term risk. Calibration and discrimination in the ABCD2 study, however, did not differ substantially from the long-term prediction models. Comparisons with other studies cannot be made, because similar studies of long-term risk assessment in patients with TIA or ischemic are not available.
The 7 models used dichotomized variables instead of continuous variables. Perhaps with continuous variables, estimations of risk would have been more precise and discrimination may improve.22 Furthermore, the stroke-based prediction models were derived from trial populations with certain inclusion and exclusion criteria. The best-fitting model for everyday practice will probably be derived from a representative cohort; this means a consecutive stroke population without exclusion criteria.
Current prediction models for long-term prognosis in TIA and stroke patients have limited validity. Calibration seems to be adequate for most models, but discrimination between patients at high risk and patients at low risk is especially relatively poor. Improvements such as implementation of continuous variables are needed, as well as extension with stronger predictors of outcome. For now, the SPI-II and the ABCD2 prediction models seem to be most adequate. Of these 2, the ABCD2 may be preferred because of its easy applicability in clinical practice.
Sources of Funding
This study was supported by a grant from the Netherlands Heart Foundation (97.155).
- Received February 3, 2010.
- Revision received March 10, 2010.
- Accepted March 12, 2010.
Adams RJ, Chimowitz MI, Alpert JS, Awad IA, Cerqueria MD, Fayad P, Taubert KA. Coronary risk evaluation in patients with transient ischemic attack and ischemic stroke: a scientific statement for healthcare professionals from the Stroke Council and the Council on Clinical Cardiology of the American Heart Association/American Stroke Association. Circulation. 2003; 108: 1278–1290.
Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998; 97: 1837–1847.
Dippel DWJ, Koudstaal PJ. We need stronger predictors of major vascular events in patients with a recent transient ischemic attack or nondisabling stroke. Dutch TIA Trial Study Group. Stroke. 1997; 28: 774–776.
Hankey GJ, Slattery JM, Warlow CP. Transient ischaemic attacks: which patients are at high (and low) risk of serious vascular events? J Neurol Neurosurg Psychiatry. 1992; 55: 640–652.
D'Agostino RB, Wolf PA, Belanger AJ, Kannel WB. Stroke risk profile: adjustment for antihypertensive medication. The Framingham Study. Stroke. 1994; 25: 40–43.
Conroy RM, Pyorala K, Fitzgerald AP, Sans S, Menotti A, De Backer G, De Bacquer D, Ducimetiere P, Jousilahti P, Keil U, Njolstad I, Oganov RG, Thomsen T, Tunstall-Pedoe H, Tverdal A, Wedel H, Whincup P, Wilhelmsen L, Graham IM. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J. 2003; 24: 987–1003.
Pocock SJ, McCormack V, Gueyffier F, Boutitie F, Fagard RH, Boissel JP. A score for predicting risk of death from cardiovascular disease in adults with raised blood pressure, based on individual patient data from randomised controlled trials. BMJ. 2001; 323: 75–81.
Kernan WN, Viscoli CM, Brass LM, Makuch RW, Sarrel PM, Roberts RS, Gent M, Rothwell P, Sacco RL, Liu RC, Boden-Albala B, Horwitz RI. The stroke prognosis instrument II (SPI-II): A clinical prediction instrument for patients with transient ischemia and nondisabling ischemic stroke. Stroke. 2000; 31: 456–462.
The Dutch TIA Trial Study Group. Predictors of major vascular events in patients with a transient ischemic attack or nondisabling stroke. Stroke. 1993; 24: 527–531.
Hankey GJ, Slattery JM, Warlow CP. Can the long term outcome of individual patients with transient ischaemic attacks be predicted accurately? J Neurol Neurosurg Psychiatry. 1993; 56: 752–759.
van Swieten JC, Koudstaal PJ, Visser MC, Schouten HJ, van Gijn J. Interobserver agreement for the assessment of handicap in stroke patients. Stroke. 1988; 19: 604–607.
Casale PN, Devereux RB, Alonso DR, Campo E, Kligfield P. Improved sex-specific criteria of left ventricular hypertrophy for clinical and computer interpretation of electrocardiograms: validation with autopsy findings. Circulation. 1987; 75: 565–572.