| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Stroke. 2000;31:456.)
© 2000 American Heart Association, Inc.
Original Contributions |
From the Departments of Internal Medicine (W.N.K., C.M.V., R.I.H.), Epidemiology and Public Health (L.M.B., R.W.M., R.I.H.), Neurology (L.M.B.), Psychiatry (P.M.S.), and Obstetrics and Gynecology (P.M.S.), Yale University School of Medicine, New Haven, Conn; Department of Clinical Epidemiology and Biostatistics, Faculty of Health Sciences, McMaster University (R.S.S., M.G.) and Clinical Trials Methodology Group, Hamilton Civic Hospitals Research Centre (M.G.), Hamilton, Ontario, Canada; Department of Clinical Neurology, University of Oxford, England (P.R.); and Neurological Institute, Columbia University School of Medicine, New York, NY (R.L.S., R-C.L., B.B-A.).
| Abstract |
|---|
|
|
|---|
MethodsTo validate SPI-I, we applied it to 4 test cohorts and calculated pooled outcome rates. To create SPI-II, we incorporated new predictive variables identified in 1 of the test cohorts and validated it in the other 3 cohorts.
ResultsFor SPI-I, pooled rates (all 4 test cohorts) of stroke or death within 2 years in risk groups I, II, and III were 9%, 17%, and 24%, respectively (P<0.01, log-rank test). SPI-II was created by adding congestive heart failure and prior stroke to SPI-I. Each patients risk group was determined by the total score for 7 factors: congestive heart failure (3 points); diabetes (3 points); prior stroke (3 points); age >70 years (2 points); stroke for the index event (not transient ischemic attack) (2 points); hypertension (1 point); and coronary artery disease (1 point). Risk groups I, II, and III comprised patients with 0 to 3, 4 to 7, and 8 to 15 points, respectively. For SPI-I, pooled rates (3 cohorts excluding the SPI-II development cohort) of stroke or death within 2 years in risk groups I, II, and III were 9%, 17%, and 23%, respectively. For SPI-II, pooled rates were 10%, 19%, and 31%, respectively. In receiver operator characteristic analysis, the area under the curve was 0.59 (95% CI, 0.57 to 0.60) for SPI-I and 0.63 (95% CI, 0.62 to 0.65) for SPI-II, confirming the better performance of the latter.
ConclusionsCompared with SPI-I, SPI-II achieves greater discrimination in outcome rates among risk groups. SPI-II is ready for use in research design and may have a role in patient counseling.
Key Words: cerebral infarction cerebral ischemia, transient cerebrovascular disorders prognosis randomized controlled trials
| Introduction |
|---|
|
|
|---|
In an effort to resolve the controversy about SPI-I and to provide further evidence on its reliability, we report in this article the performance of SPI-I in 4 independent cohorts. In addition, we report the results of an effort to revise and improve SPI-I. By adding new predictive variables and recalculating point scores for all component variables, we have created SPI-II and demonstrated its enhanced performance in 3 test cohorts.
| Subjects and Methods |
|---|
|
|
|---|
Retesting the Performance of SPI-I
For the current research, SPI-I was retested in 4 cohorts that
were selected to provide a full test of external validity and
transportability.4 Transportability refers to the
performance of a prognosis instrument in cohorts other than the
one in which it was developed. The first test cohort was the Womens
Estrogen for Stroke Trial (WEST).5 The WEST is an ongoing
randomized trial that had recruited 525 patients at the time of this
research. SPI-I was used to stratify randomization in this trial, which
is being conducted by several of the authors of this article. The other
cohorts were from the United Kingdom Transient Ischemic Attack
(UK-TIA) Aspirin Trial,6 the Clopidogrel Versus Aspirin in
Patients at Risk of Ischemic Events (CAPRIE)
Trial,7 and the Northern Manhattan Stroke Study
(NoMaSS).8 Because the WEST cohort was most similar to the
SPI-I cohort in terms of geography and methodology, we expected that
SPI-I would have its best performance in the WEST. The other
cohorts were very different from the SPI-I cohort, especially in terms
of research methodology used to define baseline variables, and
constituted a very rigorous test of transportability.
For each of the 4 cohorts separately, we calculated cumulative rates for stroke or death within 2 years using Kaplan-Meier estimates for study participants classified by SPI-I as having low, medium, or high risk.9 The log-rank test was used to evaluate differences among time-to-event curves.10
For classification of variables in SPI-I among WEST participants, we used the same criteria specified in the original validation study.2 Diabetes was classified from self-report. Hypertension was defined by blood pressure measurements performed on entry into the WEST. Two readings were recorded from each arm. Severe hypertension was present if the average of any unilateral pair of systolic readings was >180 mm Hg or if the average of any unilateral pair of diastolic readings was >100 mm Hg. The distinction between stroke and TIA was determined by the presence of symptoms lasting >24 hours. Coronary heart disease was determined by self-report of a history of myocardial infarction requiring hospitalization, ECG evidence of a Q-wave myocardial infarction, or a positive Rose Questionnaire for angina.11 Data on variables required for SPI-I were missing for no patients.
Classification of 3 of 5 variables in SPI-I was similar in all 4 test cohorts: age, the distinction between stroke and TIA for the entry event, and diabetes. Hypertension, however, was classified by a home measurement 3 months after discharge for the WEST (>180 mm Hg systolic or >100 mm Hg diastolic), by hospital measurement for NoMaSS (>180 mm Hg systolic or >100 mm Hg diastolic), by office measurement within 3 months of the index event for the UK-TIA trial, and by self-reported history for CAPRIE. Coronary artery disease was classified by self-reported history of myocardial infarction or ECG criteria for the WEST and NoMaSS, by self-report alone for CAPRIE, and by physician report for the UK-TIA cohort.
Development of an Improved Stroke Prognosis Instrument
(SPI-II)
The second goal of our analysis was to improve SPI-I. We
used data from the WEST cohort to identify new predictive variables
to add to SPI-I.
New candidate variables were identified from several sources. Many were selected because they have been incorporated in other successful prediction instruments.3 12 These include left ventricular hypertrophy (LVH), peripheral arterial disease, previous stroke or TIA, and the presence of a visible ischemic lesion on brain imaging. On the basis of other published research, we also examined the predictive effect of atrial fibrillation,13 14 15 16 17 18 carotid stenosis >80%,3 19 heart failure,20 21 22 23 and anterior location of myocardial infarction.24 25 Finally, on the basis of our own hypotheses, we examined physical performance status and cognition.
LVH was defined by the Sokolow-Lyon criteria26 and
separately by a newer sex-specific criterion.27
Peripheral arterial disease, prior stroke, and
prior TIA were defined by patient self-report of a physician diagnosis
in a structured interview. Atrial fibrillation was classified by the
presence of this rhythm on a baseline ECG or by self-report of a
history of the condition. All cases of self-reported atrial
fibrillation (with a normal ECG) required confirmation by other medical
record documentation. Heart failure was defined by a positive
answer by the patient to the question, "Have you ever had shortness
of breath or fatigue which your doctor said was due to heart trouble
(this is also called heart failure or congestive heart failure)?"
Anterior myocardial infarction was defined by the presence of Q waves
lasting
0.03 seconds in at least 2 of precordial leads
V1, V2, and
V3. Physical performance was measured
with 2 validated instruments.28 29 Cognition was measured
with the Folstein MiniMental State Examination.30 Data
were available for >98% of patients for all variables except LVH
(available in only 93%), anterior myocardial infarction (available in
only 94%), and carotid stenosis (available in only 86%).
For several variables (ie, age, physical functioning, prior cerebrovascular event, hypertension, coronary artery disease, atrial fibrillation, and LVH), we examined several alternative measures. For example, LVH was examined according to both the Cornell27 and Sokolow-Lyon criteria.26 To select one among alternative measures to include in a multivariate model, we considered the magnitude of the relative risk (RR) and the statistical significance of the RR. When these criteria did not clearly indicate that one measure was better than another, we kept the original variable definition from SPI-I.
New features were selected for inclusion in the revised instrument if
the P value for its coefficient was
0.1 in a
multivariate model. To create the new prognostic
system, SPI-II, we assigned points to each variable (the newly
selected ones plus the variables in SPI-I) based strictly on the
relative magnitude of its regression coefficient in a Cox proportional
hazards analysis, with 3 points being the maximum. For each
patient in the WEST, we summed the total point score. Risk groups were
determined by examining outcome rates in groups of patients with
distinct total point scores. Scores demarcating risk groups were
selected to combine groups of patients with similar outcome rates.
Validation of SPI-II
To validate SPI-II, we examined its performance in
cohorts from the NoMaSS, the UK-TIA Aspirin Trial, and CAPRIE. As a
summary measure of the performances of SPI-I and SPI-II, we
calculated pooled, weighted estimates of outcome rates in each risk
group using EpiMeta software (Klemm Analysis Group, Version
1.1, Centers for Disease Control). Because of significant
heterogeneity among studies for outcome rates by risk
groups, random effects model estimates are presented. To
compare directly the performance of SPI-I and SPI-II, we
calculated the area under their receiver operating characteristic (ROC)
curves using a spreadsheet program31 and Lotus 12-3
Release 4 for Windows (Lotus Development Corp).
Other data analyses were performed with SAS software (SAS Institute Inc) or SPSS (SPSS Inc). All reported probability values are 2-sided. The WEST was approved by institutional review boards at all participating hospitals, and all subjects gave informed consent.
| Results |
|---|
|
|
|---|
|
The pooled estimates across all 4 cohorts were 9%, 17%, and 24% for risk groups I, II, and III, respectively (P<0.01 for linear trend).
Development of SPI-II
Selection of Variables
For an analysis based on the WEST cohort, unadjusted and
adjusted RRs for the occurrence of stroke or death within 2 years are
listed in Table 2
for all baseline
variables. At least 1 definition of the following 6 features was
associated with risk for stroke or death in the unadjusted (bivariate)
analysis (P
0.05, log-rank test): physical
functioning, prior cerebrovascular event, coronary artery
disease, congestive heart failure, atrial fibrillation, and diabetes
mellitus.
|
In a multivariate model, 4 of 13 features were
significantly associated with stroke or death (P
0.1): age
>70 years (RR=1.6, P=0.04); prior stroke (RR=1.9,
P=0.004); heart failure (RR=1.5, P=0.1); and
diabetes (RR=1.7, P=0.009). Carotid artery disease was not
examined in the multivariate model because data were
missing for 74 patients (14%).
Creation of SPI-II
Table 3
displays the regression
coefficients from a Cox proportional hazards model and point
assignments for variables in SPI-II. Congestive heart failure,
diabetes, and prior stroke were each assigned 3 points; age >70 years
and stroke (rather than TIA) for the index event were each assigned 2
points; and severe hypertension and coronary artery disease
were each assigned 1 point.
|
The performance of SPI-II in the WEST cohort is shown in Table 4
. The 2-year outcome percentage of
stroke or death in the first risk group is slightly lower than in SPI-I
(9% compared with 11%), but the 2-year percentage in the high-risk
group is now 42% compared with 31% originally. The distribution of
patients among groups is also more even. By the log-rank test, the
trend of rates between risk groups remains statistically significant,
with a P value of <0.001.
|
Validation of SPI-II
The performance of SPI-II in 3 independent populations is
also shown in Table 4
. Most patients in the UK-TIA Aspirin Trial
(91%) were in risk group I. Percentages of stroke or death within 2
years in risk groups I, II, and III were 11%, 19%, and 17%,
respectively. There were only 6 patients in group III, however, making
the estimate of 17% unstable. The trend in rates across risk groups is
statistically significant by the log-rank test (P=0.001).
For the CAPRIE cohort, 2-year outcome percentages for risk groups I,
II, and III were 9%, 17%, and 26%, respectively
(P<0.0001, log-rank test). For NoMaSS, the outcome
percentages for risk groups I, II, and III were 16%, 24%, and 42%,
respectively (P=0.0004, log-rank test).
To compare the performances of SPI-I and SPI-II, Table 5
displays the outcome rates in each risk
stratum for each prognosis system pooled across 3 test cohorts (NoMaSS,
UK-TIA, and CAPRIE). The WEST cohort is not included in this
analysis because it was the development cohort for SPI-II. For
SPI-I, the pooled rates in risk groups I, II, and III are 9%, 17%,
and 23%, respectively (P<0.01 for linear trend). For
SPI-II, the pooled rates in risk groups I, II, and III are 10%, 19%,
and 31%, respectively (P<0.01 for linear trend). The
results show that, compared with SPI-I, SPI-II places more patients in
the low-risk group and achieves a larger risk increment between risk
groups I and III (14% compared with 21%). The values for area under
the ROC curve for SPI-I and SPI-II (for the pooled analysis)
are 0.59 (95% CI, 0.57 to 0.60) and 0.63 (95% CI, 0.62 to 0.65),
respectively.
|
| Discussion |
|---|
|
|
|---|
To improve SPI-I, we built SPI-II by including prior stroke and congestive heart failure. Prior stroke is a well-documented prognostic factor for patients with TIA or stroke.3 12 13 32 Heart failure has been determined to increase risk for recurrent stroke20 21 23 or death22 in some studies but not others.318,33 With the inclusion of prior stroke and congestive heart failure, SPI-II includes most conveniently obtained clinical variables that are well documented to influence the risk for recurrent stroke or death among patients with symptomatic cerebrovascular disease.
Testing of the revised Stroke Prognosis Instrument (SPI-II) in 3
external cohorts confirmed that SPI-II performs better than SPI-I. In
the pooled analysis, outcome rates for SPI-II in risk groups I,
II, and III were 10%, 19%, and 31%, respectively (Table 5
).
Patients were more evenly distributed among risk groups in SPI-II than
in SPI-I, and the absolute risk distinction between group I and group
III was larger for SPI-II (14% compared with 21%). To compare SPI-I
and SPI-II with a statistical measure, we calculated the areas under
ROC curves. The area was larger for SPI-II than for SPI-I, confirming
the better performance of the former.
To demonstrate transportability, we tested SPI-II in 3 cohorts with no foreknowledge of its performance in any of them. We believe that this strategy constituted a formidable test for SPI-II and showed that it transports reasonably well. In CAPRIE and the UK-TIA cohorts, outcome rates in risk groups I and II were similar to rates in the WEST cohort, indicating excellent calibration. Outcome rates in group III, however, were lower than in the WEST. In the NoMass test cohort, the opposite pattern is seen. Calibration is excellent for groups II and III, but the outcome rate was higher for NoMass group I compared with the WEST. Our findings illustrate that the performance (especially calibration) of a prognosis instrument may be affected by how patients are selected for the cohorts in which it is developed and applied.4 The lower outcome rate in group III from CAPRIE and UK-TIA cohorts may be a consequence of the better overall health and younger age of patients entering those compared with the WEST. The higher outcome rate in group I from NoMass may be a consequence of comorbid illness among the unselected hospital patients who entered this cohort compared with the WEST.
SPI-II had its best performance in the WEST (development) cohort. Beyond patient selection discussed above, we believe that 2 other factors may explain why SPI-II performs better in the WEST cohort than in the other 3. First, the multivariable model used to build SPI-II was fit to the WEST cohort. The superior performance of SPI-II in the WEST cohort is therefore an expected consequence of the analytic strategy used to construct it.34 Second, baseline variables were acquired by different means and classified according to different criteria in each of the 4 cohorts used to test SPI-II. In some of the cohorts, some variables were defined by self-report or other casual inquiry. Inconsistent or inaccurate characterization of baseline variables can impair the performance of a prognosis instrument.4 34 As evidence, the good performance of SPI-I in the WEST cohort may be explained by the fact that WEST variables were defined according to the precise criteria for the SPI-I development set. Testing of SPI-I and SPI-II in cohorts with inconsistent variable characterization allowed us to test transportability to cohorts assembled and characterized with distinct research methods.
Now that SPI-II has been validated in 3 independent cohorts, we believe that it is ready for use in clinical research and possibly ready for use in clinical care. For research, SPI-II can be used to stratify randomization in clinical trials. Stratification can reduce the chance of important imbalances between treatment groups for small (<400 patients) trials and may increase the efficiency of both small and large trials.35 SPI-I has already been used successfully to stratify randomization in the WEST.5 SPI-II may also be used to examine treatment effects within randomized controlled trial subgroups defined by prognosis. As an example, a subgroup analysis from the North American Symptomatic Carotid Endarterectomy Trial showed that surgery was associated with a larger RR reduction for ipsilateral stroke among high-risk patients compared with low-risk patients (77% compared with 48%, respectively).36 Beyond its use for research, SPI-II may be of use for patient counseling. For patients who ask to know their prognosis, SPI-II may provide reassurance for low-risk patients. For higher-risk patients, SPI-II may provide motivation to engage in risk reduction. Until SPI-II is shown to perform as well in population-based cohorts as it does in hospital-based (NoMass) and randomized controlled trial cohorts (WEST, CAPRIE, and UK-TIA), it should be applied only cautiously to individual patients in nonresearch settings.
In addition to SPI-I and SPI-II, 2 other instruments have been published and validated specifically to estimate long-term prognosis for patients with ischemic symptoms.3 12 One system, developed by Hankey and colleagues,3 has the form of 3 equations that estimate the probability that an individual patient will be free of various vascular events (stroke; coronary event; stroke, myocardial infarction, or vascular death) at 1 and 5 years. During external validation in the UK-TIA cohort, the authors of this instrument found, as we did in that same validation cohort, that patients estimated by the instrument to be at high risk did not have high outcome rates. The area under the curve for this instrument was 0.65 (95% CI, 0.62 to 0.68), very similar to the area under the curve for SPI-II (0.63; 95% CI, 0.62 to 0.65). Overall, the performances of SPI-II and the system of Hankey et al are quite similar. The reason to use one over the other may be convenience (SPI-II may be easier to apply) and outcomes of interest. The second instrument, by van Latum and colleagues,12 was developed to predict major vascular events among patients with a TIA or stroke and with atrial fibrillation. The instrument was developed in the placebo group of a randomized controlled trial and applied to the active treatment groups. Thorough performance data were not published. This instrument has not received enough testing to determine its clinical role.
Our research on SPI-I and SPI-II has 3 main limitations. First, both instruments omit stroke type (eg, lacunar infarction, embolic infarction)37 38 and aortic plaque,39 2 baseline variables that may influence prognosis but that were not available in the WEST database. Inclusion of these and other as yet unidentified predictive features may improve the performance of prognosis instruments. Second, our instruments were developed and tested in research cohorts. How they transport to nonresearch cohorts is unknown. Third, SPI-I and SPI-II were applied retrospectively to most of our test cohorts (SPI-I was applied prospectively to the WEST cohort). We believe that they both will perform optimally only with prospective application that allows optimal baseline variable classification.
Progress in clinical prediction for stroke patients will come by identifying clinical features that sharply discriminate between those who suffer a subsequent vascular event and those who do not and by applying appropriate research methods for the development of multivariable instruments. SPI-II includes most clinical features of known predictive importance and was developed with rigorous research methods. It is convenient to use, compares favorably with available prediction instruments, and targets a distinctly important clinical outcome. Although SPI-II represents progress, like all prediction instruments it will need to be reinvented or recalibrated periodically to reflect how therapy and temporal trends in population health affect disease outcome.
| Acknowledgments |
|---|
| Footnotes |
|---|
Received September 21, 1999; revision received November 5, 1999; accepted November 5, 1999.
| References |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
A. Ois, M. Gomis, A. Rodriguez-Campello, E. Cuadrado-Godia, J. Jimenez-Conde, C. Pont-Sunyer, G. Cuccurella, and J. Roquer Factors Associated With a High Risk of Recurrence in Patients With Transient Ischemic Attack or Minor Stroke Stroke, June 1, 2008; 39(6): 1717 - 1721. [Abstract] [Full Text] [PDF] |
||||
![]() |
F. Purroy, C. A. Molina, J. Montaner, and J. Alvarez-Sabin Absence of Usefulness of ABCD Score in the Early Risk of Stroke of Transient Ischemic Attack Patients Stroke, March 1, 2007; 38(3): 855 - 856. [Full Text] [PDF] |
||||
![]() |
M. S.V. Elkind, A. C. Flint, R. R. Sciacca, and R. L. Sacco Lipid-lowering agent use at ischemic stroke onset is associated with decreased mortality Neurology, July 26, 2005; 65(2): 253 - 258. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. S.V. Elkind Implications of stroke prevention trials: Treatment of global risk Neurology, July 12, 2005; 65(1): 17 - 21. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. L. Sacco, J. Sivenius, and H.-C. Diener Efficacy of Aspirin Plus Extended-Release Dipyridamole in Preventing Recurrent Stroke in High-Risk Populations Arch Neurol, March 1, 2005; 62(3): 403 - 408. [Abstract] [Full Text] [PDF] |
||||
![]() |
W. N. Kernan, C. M. Viscoli, S. E. Inzucchi, L. M. Brass, D. M. Bravata, G. I. Shulman, and J. C. McVeety Prevalence of Abnormal Glucose Tolerance Following a Transient Ischemic Attack or Ischemic Stroke Arch Intern Med, January 24, 2005; 165(2): 227 - 233. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Alamowitch, M. Eliasziw, H. J.M. Barnett, and for the North American Symptomatic Carotid Endarte The Risk and Benefit of Endarterectomy in Women With Symptomatic Internal Carotid Artery Disease Stroke, January 1, 2005; 36(1): 27 - 31. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Daffertshofer, O. Mielke, A. Pullwitt, M. Felsenstein, and M. Hennerici Transient Ischemic Attacks Are More Than "Ministrokes" Stroke, November 1, 2004; 35(11): 2453 - 2458. [Abstract] [Full Text] [PDF] |
||||
![]() |
Z. G. Nadareishvili, H. Li, V. Wright, D. Maric, S. Warach, J. M. Hallenbeck, J. Dambrosia, J. L. Barker, and A. E. Baird Elevated pro-inflammatory CD4+CD28- lymphocytes and stroke recurrence and death Neurology, October 26, 2004; 63(8): 1446 - 1451. [Abstract] [Full Text] [PDF] |
||||
![]() |
F. Purroy, J. Montaner, A. Rovira, P. Delgado, M. Quintana, and J. Alvarez-Sabin Higher Risk of Further Vascular Events Among Transient Ischemic Attack Patients With Diffusion-Weighted Imaging Acute Ischemic Lesions Stroke, October 1, 2004; 35(10): 2313 - 2319. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. Fox Poststroke patients: implications for the cardiologist Eur. Heart J. Suppl., July 1, 2003; 5(suppl_E): E4 - E10. [Abstract] [PDF] |
||||
![]() |
T G Clark, M F G Murphy, and P M Rothwell Long term risks of stroke, myocardial infarction, and vascular death in "low risk" patients with a non-recent transient ischaemic attack J. Neurol. Neurosurg. Psychiatry, May 1, 2003; 74(5): 577 - 580. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. M. Bravata, S.-Y. Ho, L. M. Brass, J. Concato, J. Scinto, and T. P. Meehan Long-Term Mortality in Cerebrovascular Disease Stroke, March 1, 2003; 34(3): 699 - 704. [Abstract] [Full Text] [PDF] |
||||
![]() |
E.J. Cunningham, R. Bond, Z. Mehta, M.R. Mayberg, C.P. Warlow, and P.M. Rothwell Long-Term Durability of Carotid Endarterectomy for Symptomatic Stenosis and Risk Factors for Late Postoperative Stroke Stroke, November 1, 2002; 33(11): 2658 - 2663. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. S.V. Elkind Implications of stroke prevention trials: Treatment of global risk Neurology, July 12, 2005; 65(1): 17 - 21. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Stroke Home | Subscriptions | Archives | Feedback | Authors | Help | AHA Journals Home | Search Copyright © 2000 American Heart Association, Inc. All rights reserved. Unauthorized use prohibited. |