Can Multivariable Risk-Benefit Profiling Be Used to Select Treatment-Favorable Patients for Thrombolysis in Stroke in the 3- to 6-Hour Time Window?
Background and Purpose— The Stroke-Thrombolytic Predictive Instrument (Stroke-TPI) uses multivariate equations to predict outcomes with and without thrombolysis. We sought to examine whether such a multivariate predictive instrument might be useful in selecting patients with a favorable risk-benefit treatment profile for therapy after 3 hours.
Methods— We explored outcomes in patients from 5 major randomized clinical trials testing intravenous recombinant tissue plasminogen activator (rt-PA) classified by the Stroke-TPI as “treatment-favorable” or “treatment-unfavorable.” We used iterative bootstrap re-sampling to estimate how such a model would perform on independent test data.
Results— Among patients treated within the 3- to 6-hour window, 67% of patients were classified by Stroke-TPI predicted outcomes as “treatment-favorable” and 33% were classified as “treatment-unfavorable.” Outcomes in the treatment-favorable group demonstrated benefit for thrombolysis (modified Rankin Scale score ≤1: 44.0% with rt-PA versus 34.2 with placebo, P=0.005), whereas harm was demonstrated in the treatment-unfavorable group (modified Rankin Scale score ≤1: 31.3% with rt-PA versus 38.3% with placebo; P=0.004). Bootstrap resampling with complete cross-validation showed that the absolute margin of benefit in the treatment-favorable group diminished on average by 36% between derivation and independent validation sets, but still represented a significant tripling of improvement in benefit compared with conventional inclusion criteria (5.2% [interquartile range, 1.7% to 8.6%] versus 1.8% [interquartile range, −0.5 to 4.1], P<0.0001).
Conclusions— Such multivariable risk-benefit profiling may be useful in the selection of acute stroke patients for rt-PA therapy even more than 3 hours after symptom onset. Prospective testing is indicated.
- acute care
- acute Rx
- acute stroke
- emergency medicine
- risk factors
- stroke management
- thrombolytic Rx
- treatment predictive modeling
Clinical trials including patients beyond the 3-hour time-window have failed to find overall benefit for intravenous recombinant tissue plasminogen activator (rt-PA) in acute stroke,1–4 although joint analysis of all randomized data strongly suggests that beneficial effects extend beyond 3 hours.5 However, treatment beyond 3 hours has not been approved by regulatory agencies or endorsed in most guidelines.6 The narrow time-window contributes to the low rate of thrombolysis.7,8
Previously, we found evidence suggesting the absence of benefit in these trials with patients enrolled beyond 3 hours from symptom onset may obscure treatment-favorable and treatment-unfavorable patients identifiable on the basis of pretreatment clinical characteristics.9 We recently developed a predictive instrument for thrombolysis in stroke (the Stroke-Thrombolytic Predictive Instrument [Stroke-TPI]), which predicts outcomes with and without thrombolysis.10 This study’s purpose was to test whether such a prognostic instrument might be useful for the selection of patients with a favorable risk-benefit profile for IV rt-PA when treated after 3 hours of stroke onset.
Overview of Analysis Design
Having developed the Stroke-TPI on a combined database of randomized clinical trials (including National Institute of Neurological Disorders and Stroke [NINDS] rt-PA Trials 1 and 2, Alteplase Thrombolysis for Acute Noninterventional Therapy in Ischemic Stroke [ATLANTIS] A and B, and European Cooperative Acute Stroke Study [ECASS] II),10 we examined whether it might be able to select treatment-favorable patients who come for treatment after 3 hours and are thereby currently ineligible for rt-PA. We used the Stroke-TPI to classify patients in the combined database into “treatment-favorable” or “treatment-unfavorable” patients based on whether they were more likely to have a normal or near-normal outcome with or without rt-PA. We examined the characteristics, outcomes, and the treatment effects in these groups.
Because the Stroke-TPI was developed on the combined database used for this analysis, these results are likely to be overly optimistic and might represent the upper limit of model performance. Therefore, we investigated how such a model would perform on independent data. Figure 1 summarizes our approach. First, the combined database was divided into derivation and test subsets. To select the derivation subset, we randomly sampled patients from the combined database with replacement (ie, bootstrap sampling), to reconstitute a derivation database that was the same size as the overall combined database (“0.632 bootstrapping”).11 Because patients are sampled with replacement, some patients are drawn repeatedly and some (≈37% on average) are excluded. The excluded portion forms the independent test dataset. Stepwise logistic regression was used on the derivation dataset to develop a model predicting the probability of a normal or near-normal outcome (modified Rankin Scale [mRS] score ≤1), with and without rt-PA, as with the development of the Stroke-TPI.10 The model was then applied to those patients treated after 3 hours in the independent test data, to classify them as “treatment-favorable” or “treatment-unfavorable,” based on model predictions. We then examined the actual outcomes in these patients, hypothesizing that “treatment-unfavorable” patients would do worse with rt-PA than with placebo, whereas “treatment-favorable” patients would benefit from rt-PA.
Because the 3- to 6-hour subset of the test dataset is not large enough to make reliable treatment-effect estimates in each of the favorable/unfavorable subgroups, and because any given sample might be nonrepresentative, we iterated the bootstrap sampling and modeling procedures 250 times. Each iteration produced a different derivation dataset, a different model (with different variables and coefficients) and a different test dataset. Through this iterative bootstrap sampling, modeling, and testing, we arrived at a stable estimate of the decrement in the degree of benefit in the test dataset compared with the derivation dataset, and also of the “gain” in the degree of benefit expected with the use of such a multivariable model in independent data compared with the conventional selection criteria.
Our primary outcome was the degree of benefit among model-selected “treatment-favorable” patients in the test set across samples, compared with that seen overall in the 3- to 6-hour sample from which the subgroup was drawn, using the mRS ≤1 outcome. In addition, as a secondary outcome, we examined the proportion of patients who were dead or severely disabled (mRS ≥5), by treat ment-assignment, in the treatment-favorable and treatment-unfavorable groups.
Overall, among patients treated between 3 and 6 hours of symptom onset, there was no significant difference in good outcomes between the treated and untreated groups at 90 days (40% in rt-PA–treated patients; 38% in control patients, P=0.51). When the Stroke-TPI model was applied to the overall database on which it was developed, the treatment-favorable group comprised 67% of the sample and demonstrated an absolute margin of benefit from rt-PA of 9.8% (mRS ≤1: 44.0% with rt-PA versus 34.2 with placebo; P<0.005). Outcomes in the 33% classified as treatment-unfavorable showed harm with rt-PA (mRS ≤1=31.3% versus 38.3% with placebo; P<0.004). The interaction for the treatment-effect between the groups was highly significant (P<0.0001).
Characteristics of High-Benefit Versus Low-Benefit Patients
Table 1 shows the characteristics of patients classified as treatment-favorable versus treatment unfavorable, according to the Stroke-TPI. There was significant overlap in the value of individual variables that modify rt-PA effect, indicating that a combination of factors needs to be examined simultaneously, as by a multivariable model. The outcomes shown in Table 1 indicate that, though the model was developed to predict a normal or near-normal outcome (mRS ≤1), patients likely to benefit on this outcome also benefitted from thrombolysis across other measures.
Iterative Bootstrap Resampling, Modeling and Testing
Across the 250 sampling iterations, each derivation dataset had 2131 patients; each test dataset had on average 784 patients. Table 2 shows the variables tested and the proportion of the 250 models into which they were selected.
When the models were applied directly to the dataset on which they were developed (Figure 2), on average, 33% (interquartile range [IQR], 27% to 39%) of patients were classified as “treatment-unfavorable,” and 67% (IQR, 61% to 72%) were classified as “treatment-favorable.” As predicted, patients in the treatment-unfavorable group, on average, had worse outcomes with rt-PA than with placebo, by an absolute margin of 10.2% (IQR, 6.9% to 14.2%), whereas patients in the treatment-favorable group, on average, had better outcomes with rt-PA than with placebo, by an absolute margin of 8.1% (IQR, 5.8% to 10.7%). When the models are applied to the test datasets (Figure 2), on average, 33% (IQR, 26% to 40%) of patients were classified as “treatment-unfavorable” and 67% (IQR, 61% to 73%) of patients were classified as “treatment-favorable.” Within the treatment-unfavorable group, again, outcomes were worse with rt-PA treatment than with placebo, whereas in the treatment-favorable group, on average, outcomes were better with rt-PA therapy. As expected, there was some “shrinkage” of the absolute margin benefit in the “treatment-favorable” group on the test datasets compared with the derivation datasets, but 64% of the margin of benefit was maintained, with an average absolute margin of 5.2% on average (IQR, 1.7% to 8.6%). This compares favorably to the overall 1.8% (IQR, −0.5 to 4.1) average treatment benefit in the 3- to 6-hour window in the test datasets from which these patients were selected (P<0.0001 for the difference in benefit between these 2 samples in the test datasets). If the benefit found using the Stroke-TPI has a similar degree of “shrinkage” when tested on independent data, than the margin of absolute benefit in the treatment-favorable subgroup (9.8% in the combined database) would be expected to be 6.3%.
Death or Severe Disability
We also examined the likelihood of severe disability or death (mRS ≥5) in patients treated in the 3- to 6-hour period, overall, and in the treatment-favorable and treatment-unfavorable subgroups. These results, shown in Figure 3, were virtually identical in the derivation and test sets, suggesting good generalizability, and suggesting that the model might be especially useful for excluding patients who would be harmed by rt-PA.
According to Stroke-TPI predictions, there is tremendous heterogeneity in the expected benefit for patients treated with rt-PA between 3 and 6 hours after stroke onset. When patients are classified as treatment-favorable versus treatment-unfavorable based on Stroke-TPI predictions, there is a dramatic divergence of treatment effects. Models developed on bootstrapped samples by automated procedures yielded somewhat less divergent treatment effects on average in treatment-favorable versus treatment-unfavorable patients (Figure 2). Yet even in the test datasets of these bootstrapped samples, the benefit seen among treatment-favorable patients represents an important improvement compared with the overall 3- to 6-hour group. These results are consistent with other recent studies emphasizing the need to look beyond the average treatment effect found in a randomized trial for therapies with significant risks and benefits.12–14
In developing the Stroke-TPI, we identified 4 variables that diminish the benefits of rt-PA: longer symptom onset to treatment time, male sex, higher systolic blood pressure, and previous stroke. The Stroke-TPI’s ability to select treatment-favorable patients in an independent population depends on whether these associations with treatment effect arose by chance in this database, or whether they are true biologically based treatment modifiers. Longer symptom onset to treatment time is well-accepted to modify the treatment effects of thrombolytic therapy, based on clinical trials,5,15 as well as biologic plausibility, animal models of reperfusion16 and by analogy with results in acute myocardial infarction (AMI).17 There is ample evidence from independent data that higher systolic blood pressure is associated with an increased risk of thrombolytic-related intracranial hemorrhage in both AMI18–20 and stroke.21–23 Previous stroke has also been shown in multiple studies to increase the risk of thrombolytic-related intracranial hemorrhage in AMI,18,20 and presumably would diminish the probability of a normal or near-normal outcome even in patients who do not have hemorrhagic complications of thrombolysis. That male sex would be associated with less benefit from thrombolytics was initially unexpected. However, after the report of the sex-by-treatment interaction in this database,24 confirmatory findings from independent data have been published.25,26,27
There are several limitations to our study. First, a much larger database would have allowed for a single independent test data set sufficient for reliable treatment effect estimates in treatment-favorable and treatment-unfavorable subgroups, without the need for bootstrap re-sampling. However, as we used a complete cross-validation method (ie, new variables selected at each iteration, not just new coefficients) and validated on data excluded from model development, our test set results provide an estimate of the degree of over-optimism in the Stroke-TPI performance. The performance of the models built during the automated procedure likely represents an underestimation of the “gain” expected when using the Stroke-TPI on independent data, because the variable–outcome relationships in these models were not individually checked against clinical reasoning, and results of all models, including those of dubious quality, were averaged. In particular, although the Stroke-TPI includes 4 robust treatment effect interactions on which risk-benefit profiling depend, 56 of the 250 bootstrapped models included ≤2 treatment–effect interactions, including 23 that did not even include a treatment-by-time interaction. Additionally, ongoing trials (such as ECASS 3) should provide the opportunity for future independent testing.
The model we present was limited also by the variables collected commonly across trials. In particular, serum markers and perfusion-diffusion magnetic resonance imaging have shown some preliminary promise for risk-benefit profiling, but these data were not included in these trials.
Despite these limitations, the approach presented here could be of substantial use. In the context of a clinical trial, improving the margin of benefit from 1.8% to 5.2% would dramatically diminish the sample size required for an adequately powered clinical trial, from an infeasible >20 000 to a more feasible <3000 patients. (With an expected margin of benefit of 6.3%, ≈1000 patients would be required for each trial arm for 80% statistical power.) Use of such a model could also have an important effect in clinical care. In the ECASS I and II trials, ≈80% to 85% of patients were treated after 3 hours. Thus, using estimates from these enrollees, treating even two-thirds of this group of patients could more than triple the number of patients eligible for rt-PA (ie, from 20% of ECASS enrollees to >70%), although other studies suggest that the number of patients otherwise eligible for treatment in this time window may be less.28
In summary, we have demonstrated that there is a great deal of heterogeneity in the treatment effect of rt-PA for acute stroke, and that the Stroke-TPI may be effective in identifying those patients most likely to benefit from treatment in the 3- to 6-hour window. This approach requires testing in a prospective trial.
The authors would like to acknowledge the work of the NINDS, ATLANTIS, and ECASS Trialists, without which this secondary analysis would not be possible.
Sources of Funding
This research was supported by grants from the NINDS to Dr Kent (K23NS44929 and R21NS48225).
E.B. is an employee at Boehringer Ingelheim, which has a financial interest in this study. W.H. has received honoraria for speaking at major symposia for Boehringer Ingelheim, and for participating in DSMB and a steering committee for Boehringer Ingelheim. W.H. is also on an advisory board for Boehringer Ingelheim. All other authors report no conflicts of interest.
- Received May 19, 2006.
- Accepted June 21, 2006.
Hacke W, Kaste M, Fieschi C, Toni D, Lesaffre E, von Kummer R, Boysen G, Bluhmki E, Hoxter G, Mahagne MH, Hennerici M. Intravenous thrombolysis with recombinant tissue plasminogen activator for acute hemispheric stroke. The European Cooperative Acute Stroke Study (ECASS). JAMA. 1995; 274: 1017–1025.
Hacke W, Kaste M, Fieschi C, von Kummer R, Davalos A, Meier D, Larrue V, Bluhmki E, Davis S, Donnan G, Schneider D, Diez-Tejedor E, Trouillas P. Randomised double-blind placebo-controlled trial of thrombolytic therapy with intravenous alteplase in acute ischaemic stroke (ECASS II). Second European-Australasian Acute Stroke Study Investigators. Lancet. 1998; 352: 1245–1251.
Clark WM, Wissman S, Albers GW, Jhamandas JH, Madden KP, Hamilton S. Recombinant tissue-type plasminogen activator (Alteplase) for ischemic stroke 3 to 5 hours after symptom onset. The ATLANTIS Study: a randomized controlled trial. Alteplase Thrombolysis for Acute Noninterventional Therapy in Ischemic Stroke. JAMA. 1999; 282: 2019–2026.
Clark WM, Albers GW, Madden KP, Hamilton S; Thromblytic therapy in acute ischemic stroke study investigators. The rt-PA (alteplase) 0- to 6-hour acute stroke trial, part A (A0276g): results of a double-blind, placebo-controlled, multicenter study. Stroke. 2000; 1: 811–816.
Adams HP Jr, Adams RJ, Brott T, del Zoppo GJ, Furlan A, Goldstein LB, Grubb RL, Higashida R, Kidwell C, Kwiatkowski TG, Marler JR, Hademenos GJ. Guidelines for the early management of patients with ischemic stroke: a scientific statement from the Stroke Council of the American Stroke Association. Stroke. 2003; 34: 1056–1083.
Kent DM, Ruthazer R, Selker HP. Are some patients likely to benefit from recombinant tissue-type plasminogen activator for acute ischemic stroke even beyond 3 hours from symptom onset? Stroke. 2003; 34: 464–467.
Kent DM, Selker HP, Ruthazer R, Bluhmki E, Hacke W. The Stroke-TPI: A predictive instrument for intravenous thrombolysis in acute ischemic Stroke. 2006; 37: 2957–2962.
Hayward RA. Kent DM. Vijan S. Hofer TP. Reporting clinical trial results to inform providers, payers, and consumers. Health Affairs. 2005; 24: 1571–1581.
Marler JR, Tilley BC, Lu M, Brott TG, Lyden PC, Grotta JC, Broderick JP, Levine SR, Frankel MP, Horowitz SH, Haley EC Jr, Lewandowski CA, Kwiatkowski TP. Early stroke treatment associated with better outcome: the NINDS rt-PA Stroke study. Neurology. 2000; 55: 1649–1655.
Guyot AM, Levine SR. Pre-clinical testing of thrombolytic therapy for stroke. Lyden PD, ed. In: Thrombolytic Therapy for Stroke. Totowa, NJ: Humana Press Inc; 2001: 65–74.
Selker HP, Beshansky JR, Schmid CH, Griffith JL, Longstreth WT Jr, O’Connor CM, Caplan LR, Massey EW, D’Agostino RB, Laks MM, Lee KL, Maynard C, Wagner GS, Weaver WD, Califf RM. Presenting pulse pressure predicts thrombolytic therapy-related intracranial hemorrhage: Thrombolytic Predictive Instrument (TPI) Project results. Circulation. 1994; 90: 1657–1661.
Brass LM, Lichtman JH, Wang Y, Gurwitz JH, Radford MJ, Krumholz HM. Intracranial hemorrhage associated with thrombolytic therapy for elderly patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project. Stroke. 2000; 31: 1802–1811.
Larrue V, Von Kummer RR, Muller A, Bluhmki E. Risk factors for severe hemorrhagic transformation in ischemic stroke patients treated with recombinant tissue plasminogen activator: a secondary analysis of the European-Australasian Acute Stroke Study (ECASS II). Stroke. 2001; 32: 438–441.
Levy DE, Brott TG, Haley EC, Marler J. Factors related to intracranial hematoma formation in patients receiving tissue-type plasminogen activator for acute ischemic stroke. Stroke. 1994; 25: 291–297.
Kidwell CS, Saver JL, Carneado J, Sayre J, Starkman S, Duckwiler G, Gobin YP, Jahan R, Vespa P, Villablanca JP, Liebeskind DS, Vinuela F. Predictors of hemorrhagic transformation in patients receiving intra-arterial thrombolysis. Stroke. 2002; 33: 717–724.
Kent DM, Price LL, Ringleb P, Hill MD, Selker HP. Sex-based differences in response to recombinant tissue plasminogen activator in acute ischemic stroke: a pooled analysis of randomized clinical trials. Stroke. 2005; 36: 62–65.
Savitz SI. Schlaug G. Caplan L. Selim M. Arterial occlusive lesions recanalize more frequently in women than in men after intravenous tissue plasminogen activator administration for acute stroke. Stroke. 2005; 36: 1447–1451.
Saposnik G. Di Legge S. Webster F. Hachinski V. Predictors of major neurologic improvement after thrombolysis in acute stroke. Neurology. 2005; 65: 1169–1174.
Hill MD, Kent DM, Hinchey J, Rowley H, Buchan AM, Wechsler LR, Higashida RT, Fischbein NJ, Dillon WP, Gent M, Firszt CM, Schulz GA, Furlan AJ; PROACT-2 Investigators. Sex-based differences in the effect of intra-arterial treatment of stroke: analysis of the PROACT-2 study. Stroke. 2006; 37: 2322–2325.
California Acute Stroke Pilot Registry (CASPR) Investigators, Prioritizing interventions to improve rates of thrombolysis for ischemic stroke. Neurology. 2005; 64: 654–659.