Validating the Stroke-Thrombolytic Predictive Instrument in a Population in the United Kingdom
Background and Purpose—This study aimed to test the explanatory qualities of the Stroke-Thrombolytic Predictive Instrument (S-TPI) when applied to patients treated in routine practice.
Methods—S-TPI predictions were compared with observed outcomes in terms of normal/near-normal (modified Rankin Scale score, ≤1) and catastrophic outcome (modified Rankin Scale score, ≥5) at 3 months. Logistic regression was used to calibrate and expand the S-TPI.
Results—The S-TPI overestimated probability of catastrophic outcomes and overestimated the probability of a normal/near normal outcome above 0.4 and underestimated those below. Calibrating the S-TPI minimized discrepancies between predicted and observed outcomes, in the case of normal/near-normal outcomes, where including additional predictors (serum glucose and signs of current infarction on pretreatment brain scan) further reduced discrepancies between predicted and observed outcomes.
Conclusions—The explanatory power of the S-TPI in thrombolytic-treated patients can be improved to reflect outcomes seen in routine practice.
Predictive equations are useful to support clinical decision-making about thrombolysis with recombinant tissue plasminogen activator in acute stroke and to communicate risk/benefit information to patients and families.1 The Stroke-Thrombolytic Predictive Instrument2 (S-TPI) provides patient-specific predictions at 3 months for the likelihood of a normal/near-normal outcome (modified Rankin scale score, ≤1: no symptoms or slight disability), referred to as a normal outcome hereafter, and of catastrophic outcome (modified Rankin Scale score ≥5: severe disability/death).
A single-center cohort study (N=301) reported the S-TPI had reasonable external validity when applied to patients treated in routine practice but overestimated and underestimated probabilities for normal and catastrophic outcomes, respectively.3 We aimed to identify sources of prediction discrepancies between the S-TPI and outcomes in a larger population of patients treated in routine practice and to identify extensions that enhance the explanatory properties of the S-TPI.
Materials and Methods
Calibration curves were used to establish how predictions from the S-TPI corresponded with outcomes in the Safe Implementation of Treatments in Stroke United Kingdom (SITS-UK)4 population treated with recombinant tissue plasminogen activator between December 2002 and February 2010 in United Kingdom centers (N=4022).
Stepwise logistic regression was used to identify predictor variables associated with underprediction or overprediction of outcomes in SITS-UK patients. We also tested whether 3 additional patient characteristics (congestive heart failure, signs of current infarction on pretreatment brain scan, and serum glucose5) would improve the explanatory power of the model for normal outcomes in treated patients. For normal outcomes, a parsimonious method of calibration that estimated only an intercept and single calibration coefficient was rejected because of uncertainty about the differing relative strength of the predictors in the 2 datasets.6 Because the S-TPI assumes no association between treatment with recombinant tissue plasminogen activator and a catastrophic outcome, and because death before 3 months is a competing risk to a normal outcome at 3 months, only those surviving (modified Rankin Scale score, 0–5) at 3 months were used in the calibration of normal outcomes.
Receiver-operating curves were used to estimate the ability of the S-TPI to discriminate between those most and least likely to benefit from treatment, and between those most and least likely to experience a catastrophic outcome.
The characteristics of the SITS-UK patients and of those used to develop the S-TPI are shown in Table 1. The calibration of normal outcomes included 1860 cases (1583 cases were excluded because the dependency state was not recorded; 352 had died within 3 months; 123 had treatment times or systolic blood pressures outside the defined ranges; 104 were missing systolic blood pressure, glucose, or stroke severity information). The calibration of a catastrophic outcome included 2212 cases.
Original S-TPI Predictions of Outcomes in SITS-UK Data
Calibration curves for predicted probability of the S-TPI of normal and catastrophic outcomes in the SITS-UK population are shown in FigureA and B. The S-TPI underpredicts the probability of catastrophic outcomes in the SITS-UK population; for example, a predicted P=0.60 equates to an actual observed P=0.50 (Figure A). The S-TPI overpredicts the probability of normal outcomes in the SITS-UK population (Figure B). At low probabilities of normal outcome, the overprediction is reversed and the S-TPI underpredicts.
Calibration for Normal Outcomes
The parameter estimates for the calibrated S-TPI are shown in Table 2. The S-TPI prediction is retained (1.3770; P=0.0117). No prediction discrepancy is associated with diabetes, previous stroke, and systolic blood pressure. Prediction discrepancy is associated with male gender, age, and National Institutes of Stroke Scale score. Of the additional predictors, infarction on pretreatment brain scan and serum glucose are also found to be associated with a normal outcome. Figure C shows the improved areas under the curve (0.754–0.766) for the calibrated S-TPI models for all cases, including those who did not survive to 3 months, reflecting the S-TPI finding of an absence of association between treatment with recombinant tissue plasminogen activator and death.
Calibration for Catastrophic Outcomes
The SITS-UK population risk of catastrophic outcome was greater than predicted by the S-TPI (Table 2). No receiver-operating curve is shown for catastrophic outcome because the parsimonious recalibrating does not affect the ranking of case, but the area under the curve is 0.784.
Consistent with previous research, we found evidence that the S-TPI overestimates the probability of a normal outcome and underestimates the probability of a catastrophic outcome in treated patients. The strength of the calibrated S-TPI model is its applicability to current practice because the predictions are adjusted using data about patients routinely treated up to year 2010, and it includes additional patient characteristics.
In terms of weaknesses, there may have been bias in the routine practice data. For example, 1 possible reason for the overprediction of normal outcomes is that United Kingdom clinicians (compared with European/North America clinicians) may assign lower modified Rankin Scale scores to patients with similar levels of disability. Studies assessing inter-rater reliability of modified Rankin assessments show only modest agreement, with a kappa of <0.5.7 Prediction -discrepancies associated with men and additional predictors mean that untreated outcomes cannot be estimated using the calibrated model. Like the S-TPI, our model predicts no overall harm from treatment; its use as a guide for clinical decision-making is only warranted when thrombolytic treatment is considered to have no association with increased mortality (an assumption more valid at a population level than an individual level) or used with separate predictors of harmful outcomes.
Notwithstanding the assumption about the association between treatment and death, our findings suggest that recalibrated S-TPI is a good basis for predicting outcomes at 3 months in treated patients and its explanatory power can be improved to reflect outcomes seen in routine practice.
The authors express thanks to Kennedy R. Lees (Professor of Cerebrovascular Medicine, University of Glasgow) for access to the -SITS-UK data. We thank all UK sites that contributed data to the SITS-UK database.
Sources of Funding
This article presents independent research commissioned by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research scheme (RP-PG-0606-1241). The views -expressed in this publication are those of the authors and are not -necessarily those of the NHS, the NIHR, or the Department of Health.
Dr Ford’s institution has received research grants from Boehringer Ingelheim (manufacturer of Alteplase) and honoraria from Lundbeck for stroke-related activities. Dr Ford also has received personal -remuneration for educational and advisory work from Boehringer Ingelheim and Lundbeck.
- Received July 26, 2012.
- Accepted August 27, 2012.
- © 2012 American Heart Association, Inc.
- Flynn D,
- Murtagh M,
- Stobbart L,
- Rodgers H,
- Ford GA,
- Thomson RG
- Kent DM,
- Selker HP,
- Ruthazer R,
- Bluhmki E,
- Hacke W
- Uyttenboogaart M,
- Stewart RE,
- Vroomen PC,
- Luijckx GJ,
- De Keyser J
- 4.↵SITS-ISTR. Safe Implementation of Treatments in Stroke - International Stroke Thrombolysis Registry. Available at: https://sitsinternational.org/. Accessed March 4, 2011.
- Wahlgren N,
- Ahmed N,
- Eriksson N,
- Aichner F,
- Bluhmki E,
- Dávalos A,
- et al
- Quinn TJ,
- Dawson J,
- Walters MR,
- Lees KR