Abstract
Background and Purpose—It remains debated whether clinical scores can help identify acute ischemic stroke patients with large-artery occlusion and hence improve triage in the era of thrombectomy. We aimed to determine the accuracy of published clinical scores to predict large-artery occlusion.
Methods—We assessed the performance of 13 clinical scores to predict large-artery occlusion in consecutive patients with acute ischemic stroke undergoing clinical examination and magnetic resonance or computed tomographic angiography ≤6 hours of symptom onset. When no cutoff was published, we used the cutoff maximizing the sum of sensitivity and specificity in our cohort. We also determined, for each score, the cutoff associated with a false-negative rate ≤10%.
Results—Of 1004 patients (median National Institute of Health Stroke Scale score, 7; range, 0–40), 328 (32.7%) had an occlusion of the internal carotid artery, M1 segment of the middle cerebral artery, or basilar artery. The highest accuracy (79%; 95% confidence interval, 77–82) was observed for National Institute of Health Stroke Scale score ≥11 and Rapid Arterial Occlusion Evaluation Scale score ≥5. However, these cutoffs were associated with false-negative rates >25%. Cutoffs associated with an false-negative rate ≤10% were 5, 1, and 0 for National Institute of Health Stroke Scale, Rapid Arterial Occlusion Evaluation Scale, and Cincinnati Prehospital Stroke Severity Scale, respectively.
Conclusions—Using published cutoffs for triage would result in a loss of opportunity for ≥20% of patients with large-artery occlusion who would be inappropriately sent to a center lacking neurointerventional facilities. Conversely, using cutoffs reducing the false-negative rate to 10% would result in sending almost every patient to a comprehensive stroke center. Our findings, therefore, suggest that intracranial arterial imaging should be performed in all patients with acute ischemic stroke presenting within 6 hours of symptom onset.
Introduction
Mechanical thrombectomy, preceded by intravenous thrombolysis in eligible patients, is now considered standard of care for acute ischemic stroke (AIS) due to large-artery occlusion (LAO).1–3 Although thrombectomy has been shown to be superior to best medical treatment ≤6 hours after stroke onset, it should be started as early as possible because time to endovascular reperfusion is strongly associated with long-term outcome.4 Ideally, therefore, AIS patients with suspected LAO should be sent to the nearest comprehensive stroke center (CSC) with neurointerventional facilities at the earliest opportunity, whereas patients with AIS unlikely to have an LAO should be referred to primary stroke centers, hence avoiding overwhelming CSCs with inappropriate transfers.5 Several dedicated clinical scores, as well as National Institute of Health Stroke Scale (NIHSS) score cutoffs, have been proposed to help predict LAO in patients with suspected AIS and might be used for prehospital triage.6–8 However, data on the predictive values of particular scores in independent populations is scarce, and the optimal NIHSS score cutoff for LAO prediction varies greatly across studies, ranging from 6 to 14.8,9 Although a recent European consensus statement suggested that if baseline arterial imaging is not available, an NIHSS score ≥9 points (within 3 hours of stroke onset) or ≥7 points (within 6 hours) may indicate LAO,2 current US and Canadian guidelines do not state a clinical score as useful to predict LAO.3,10 It remains, therefore,debated whether clinical scores can improve patient triage. The aim of the present study was to determine, in an independent cohort of patients with AIS admitted within 6 hours of symptom onset, the accuracy of published clinical scores in predicting LAO.
Methods
Patients
We reviewed all consecutive patients admitted to our CSC within 6 hours after ischemic stroke onset, between January 2008 and June 2015. Inclusion criteria for the present study were (1) magnetic resonance angiography (MRA) or computed tomographic angiography (CTA) performed on admission; and (2) ischemic stroke proven on diffusion-weighted imaging or follow-up CT scan. Exclusion criteria were (1) unknown time of symptom onset; (2) missing admission NIHSS score items; (3) previous stroke with significant disability (modified Rankin Scale score, >1); (4) insufficient quality of MRA or CTA to assess LAO status; (5) patients transferred from a primary stroke center to receive endovascular therapy, as including such patients would lead to an unusually high prevalence of LAO.
Neurological examination, including NIHSS score assessment, was performed immediately after admission by a stroke neurologist. Intravenous thrombolysis was administered according to European Guidelines, within 4.5 hours of symptom onset. Endovascular therapy was used in selected cases using intra-arterial alteplase or thrombectomy. Time of symptom onset, sex, age, history of hypertension, diabetes mellitus, current smoking, atrial fibrillation, coronary artery disease, previous stroke, or transient ischemic attack, baseline NIHSS score, serum glucose, and blood pressure were routinely collected on admission.
In accordance with the French legislation, this study did not need approval by an ethics committee nor written informed consent from patients because it implied only analysis of anonymized data collected prospectively as part of routine clinical care.
Imaging
Brain imaging was performed immediately after neurological examination. Magnetic resonance imaging has been systematically implemented in our center as first-line diagnostic imaging in candidates for reperfusion therapies. The MR protocol, described in detail elsewhere,11 remained unchanged throughout the study period and included at least 3-dimensional time-of-flight intracranial MRA and T2*-weighted gradient echo imaging, performed on a 1.5-T GE Healthcare MR scanner.
Patients in whom magnetic resonance imaging was contraindicated underwent CT and, whenever feasible, neck and intracranial CTA (64-section multidetector scanner, LightSpeed; GE Healthcare, Chicago, IL).
The site of arterial occlusion was determined by neuroradiologists blinded to admission NIHSS scores. In patients undergoing magnetic resonance imaging MRA and T2* sequences were simultaneously analyzed to confirm the presence of an arterial occlusion. For the main analyses, LAO was defined as occlusion of the internal carotid artery, proximal (M1) segment of the middle cerebral artery, or basilar artery.6 In a sensitivity analysis, M2 occlusion was considered to be part of the definition of LAO. Finally, we performed analyses excluding patients with posterior circulation strokes.
Predictive Scores
For each patient, we retrospectively calculated the following scores, which have been previously proposed to predict LAO or assess stroke severity: modified NIHSS,12 3-item stroke scale (3I-SS),8 Rapid Arterial Occlusion Evaluation Scale (RACE),6 Cincinnati Prehospital Stroke Scale (CPSS),13 Cincinnati Prehospital Stroke Severity Scale (CPSSS),14 Maria Prehospital Stroke Scale (MPSS),15 shortened versions of the NIHSS (sNIHSS-1, sNIHSS-5, and sNIHSS-8),16 abbreviated NIHSS,17 out of hospital NIHSS,17 retrospective NIHSS profiles (A–F),18 and Recognition of Stroke in the Emergency Room (ROSIER).19
Table 1 and Table I in the online-only Data Supplement provide a detailed description of each score and outline minor modifications made to the RACE, CPSS, and ROSIER scales because few specific items were not routinely collected in our database. We tried to identify a published cutoff for every score, which usually consisted in the cutoff maximizing the sum of sensitivity and specificity to predict LAO in the derivation cohort (intersection of the sensitivity and specificity curves).6,20
Correspondence Between Published Clinical Scores and NIHSS Score Items
Statistical Analysis
Continuous variables were described as mean±SD or median (interquartile range) and categorical variables as number (percentage). We assessed the performances of published cutoffs for each clinical score to predict LAO by calculating its false-positive and negative rates (FPR and FNR), sensitivity, specificity, accuracy, positive and negative predictive values and likelihood ratios, with corresponding 95% confidence intervals. When no published cutoff was available for a given score, we used the cutoff maximizing the sum of sensitivity and specificity in our population. Because clinically it is important to exclude as few patients as possible from potentially beneficial therapy, we also determined, for each score, the cutoff associated with an FNR ≤10% in our cohort.21 Statistical analysis was performed using SAS 9.4 (SAS Institute, Inc, Cary, NC).
Results
During the study period, 1340 patients were admitted in our center within 6 hours of ischemic stroke onset, of which, 336 (25.1%) were not eligible for the present study (details are given in Figure I in the online-only Data Supplement), leaving 1004 patients for the analysis. Patients’ characteristics are summarized in Table 2. The median age and NIHSS were 72 (interquartile range, 59–82) and 7 (interquartile range, 2–15; range, 0–40), respectively. Median onset-to-admission and onset-to-imaging times were 135 (91–195) and 147 (105–230) minutes, respectively. Arterial imaging at admission consisted of MRA in 941 (93.7%) patients. Three hundred twenty-eight (32.7%) patients had an LAO (M1, internal carotid artery, or basilar artery).
Population Characteristics (n=1004)
Table 3 shows, for each predictive score, the percentage of patients above a published cutoff—the proportion of patients who would have been sent to a CSC if triage was based on this score—and the associated accuracy, FNR and FPR. FNR represents the proportion of patients who would not have been sent to a CSC despite LAO, hence delaying optimal treatment, and FPR represents the proportion of patients who would have been sent to a CSC despite lack of LAO (futile transfers). The highest accuracy (79%; 95% confidence interval, 77–82) was observed for NIHSS score ≥11 (FNR, 27% [22–32]; FPR, 17% [14–20]), NIHSS score ≥14 (FNR, 39% [34–44]; FPR, 12% [9–14]), and RACE score ≥5 (FNR, 33% [28–38]; FPR, 15% [12–17]). Positive and negative predictive values, sensitivity, specificity, and likelihood ratios for each published cutoff are presented in Table II in the online-only Data Supplement.
False-Negative Rate, False-Positive Rate, and Accuracy of Published Cutoffs for Various Clinical Scores to Predict Large-Artery Occlusion
FNRs ≤10% were observed when using the following cutoffs: 5 for NIHSS; 3 for modified NIHSS; 2 for MPSS, sNIHSS-8, and ROSIER; 1 for CPSS, sNIHSS-5, RACE, and abbreviated NIHSS; and 0 for CPSSS, 3I-SS, and sNIHSS-1 (Table III in the online-only Data Supplement).
Finally, 2 sensitivity analyses showed, as compared to the main analysis: (1) slightly higher FNRs and lower FPRs, when the definition of LAO encompassed M2 occlusions (Table IV in the online-only Supplement); and (2) slightly lower FNRs and higher FPRs, when the 183 (18.2%) patients with posterior circulation strokes were excluded (data not shown).
Discussion
In this large cohort of patients with AIS undergoing detailed clinical examination and MRA or CTA within 6 hours of symptom onset, we found that although several clinical scores showed a good accuracy to predict LAO, at least 20% of patients with LAO would be missed when applying published cutoffs. Moreover, cutoffs reducing the FNR to ≤10% were associated with an FPR of at least 45%. We were unable to identify a reliable cutoff to rule out LAO.
Effective patient triage based on reliable clinical prediction of LAO status would be useful not only in the prehospital setting but also for patients admitted to community hospitals where vascular imaging might not be readily available. The widely used NIHSS score has been consistently shown to be strongly associated with LAO at baseline.20 However, whether this score is truly helpful in identifying patients with LAO remains debated.21,22 Indeed, at intersection of sensitivity and specificity curves to identify LAO, Hansen et al9 observed an NIHSS score cutoff of 6, whereas other groups suggested very different cut points, such as 14.8 The NIHSS value maximizing the sum of sensitivity and specificity was 11 in our population, which is in line with previous studies.6,7,21 However, using this cutoff as a rule for patient triage would have led to sending 35% of our cohort to a CSC but would have resulted in a potential loss of opportunity for the 27% of patients with an LAO sent to centers lacking neurointerventional facilities. Although a recent study showed a much lower FNR (12%) using the NIHSS score cutoff ≥11,6 our results are in line with 2 previous studies in which FNRs of 23% and 35% were observed.7,21
One may argue that the cutoffs maximizing the sum of sensitivity and specificity may not be optimal in the particular setting of LAO prediction because a high FNR has greater implications than a high FPR, as the former would lead to delaying time to reperfusion, whereas the latter would only lead to an increase in the workload of CSCs. Therefore, we attempted to identify an NIHSS score cutoff that would allow practitioners to reasonably rule out LAO but were unable to find one. This is consistent with a previous large study reporting that 29% of patients with baseline NIHSS score of 0 had a proximal occlusion on CTA.22 The NIHSS score cutoff limiting the FNR to 10% was 5 in our cohort. However, its clinical relevance is questionable because it would have implied sending 60% of our cohort to a CSC, with a proportion of futile transfers of 46%.
Three additional limitations of the NIHSS score for LAO prediction have been previously reported. First, the optimal cutoff for LAO prediction decreases over time.9,20,23 This potential issue can be overcome by using different cutoffs depending on onset-to-examination time.20 However, this approach did not allow to notably reduce the FNRs in our cohort. Second, the predictive value of the NIHSS for LAO has been reported to be poor in posterior circulation strokes,20 for which published cutoffs are lower than those in anterior circulation strokes.9 Nonetheless, we decided to include both anterior and posterior circulation strokes in our main analysis because it can be difficult to distinguish these 2 types of stroke in the prehospital setting. Third, emergency medical teams are not always trained to perform the NIHSS, which has been occasionally considered unwieldy in the prehospital setting.16,24 To address this last concern, many simpler clinical scales have been proposed, the majority of these consisting of selected NIHSS items used in a simplified scoring system.6,8,13,14,17 Although most of these scales were designed to identify patients with stroke13,19 or to assess stroke severity,14 several publications suggest that such simple scales could accurately predict LAO and hence be used to improve patient triage.8,14,24 Of note, the prehospital RACE scale, based on the NIHSS items with the highest predictive value for LAO, was specifically designed to identify such patients and was recently validated in a prospective cohort of 357 patients.6 However, the respective predictive abilities of those simple clinical scales had not so far been compared in a large independent cohort.
We found that simple clinical scales share similar limitations to the NIHSS score on LAO prediction: although published cutoffs were associated with a high accuracy, especially for RACE and CPSSS, FNRs were >30%. If applied as a decision rule, cutoffs associated with an FNR ≤10% in our cohort—namely, CPSS≥1, CPSSS score ≥0, 3I-SS≥0, abbreviated NIHSS ≥1, or RACE≥1—would result in sending virtually all patients with an abnormal neurological examination to a CSC.
Altogether, our findings call into question the usefulness of a clinical score to identify the best candidates for thrombectomy. Indeed, even in the best case scenario (all patients in our cohort had a confirmed ischemic stroke and were examined by a stroke neurologist), clinical scales could not reliably identify patients with LAO. Therefore, although clinical scores may provide a rough estimate of the probability of LAO, MRA or CTA should be performed in all patients with symptomatic ischemic stroke presenting within 6 hours of symptom onset. Prehospital brain and arterial imaging using a mobile stroke unit might be an interesting option for patient triage in selected areas, but it remains to be demonstrated whether this expensive management approach is cost-effective in the era of bridging therapy.25–27 Other promising strategies that could help prehospital identification of stroke patients with LAO include serum biomarkers aiming to differentiate AIS and intracerebral hemorrhage28,29 and automated or telemedicine-guided transcranial ultrasound imaging.30,31
Our study has several limitations. First, our population does not represent patients with suspected stroke examined in the prehospital setting but those admitted to a CSC. Prehospital suspicion of LAO or neurological scores/screening tools do not influence the choice of primary receiving stroke center by emergency medical services in Paris, which is based on proximity and bed availability. Of note, 3 of 4 stroke centers located within our catchment area (South-West Paris and inner suburbs) have neurointerventional facilities. Patients initially sent to a primary stroke center and then transferred to our center to undergo endovascular therapy were not included in the present study. Second, we excluded patients with other diagnosis than ischemic stroke. This limitation, along with the first one, accounts for the higher prevalence of LAO in our cohort than would be expected in a prehospital setting. However, our findings remain valid because although a lower prevalence would lead to a lower positive predictive value and a higher negative predictive value than we observed, our work focused on FNR and FPR, which, like sensitivity and specificity, are considered to be independent of disease prevalence.32 Third, arterial status was mostly assessed using time-of-flight MRA, which may have led to misclassifying high-grade intracranial stenosis as occlusion. However, T2* sequences were simultaneously analyzed to confirm the presence of a thrombus whenever positive. In spite of this, we may also have mistaken a few chronic asymptomatic arterial occlusions for recently symptomatic occlusions. Fourth, prehospital scales were scored retrospectively based on the items of each patient’s NIHSS score immediately after admission. This approach enabled evaluation of numerous scales without delaying prehospital acute stroke management.33 However, we had to slightly modify 2 scores (CPSS and RACE) by replacing an item not routinely collected in our database with the closest NIHSS item. Similarly, only items from the ROSIER scale based on clinical examination were considered in the present study. It is unlikely that these minor modifications had a significant effect on our results. We could not assess the predictive ability of 3 other interesting scores that include several parts of the neurological examination not collected in our database.24,34,35 Two additional scales designed to identify stroke in the prehospital settings, namely Los Angeles Pre-Hospital Stroke Screen and Melbourne Ambulance Stroke Screen, could not be assessed in the present study because grip strength is not collected in our database.36 Although these scales seem to have good sensitivity and specificity for identifying patients with stroke,36 a major limitation of both scales is that patients aged >45 years are automatically excluded, which raises concerns as stroke incidence is increasing in young patients.37
In conclusion, published clinical scores cutoffs to predict LAO are associated with a high accuracy, but also with a high FNR. Using such cutoffs as a rule for patient triage would, therefore, result in a loss of opportunity for >20% of patients with LAO who would be inappropriately sent to centers lacking neurointerventional facilities. Conversely, using cutoffs limiting the FNR to 10% would result in sending virtually all patients with an abnormal neurological examination to a CSC. Thus, our findings call into question the usefulness of clinical scores to identify LAO and suggest that MRA or CTA should be performed in all patients with symptomatic ischemic stroke presenting within 6 hours of symptom onset.
Acknowledgments
We thank Robert Hurford for English language editing.
Disclosures
G. Turc is the primary investigator of an academic randomized controlled trial evaluating the cost–utility of a mobile stroke unit in France, funded by the Programme de Recherche Medico-Economique of the French Ministry of Health (Acute Stroke: Prehospital versus in-Hospital start of recanalization therapy [ASPHALT] trial). M. Tisserand was funded by the Société Française de Radiologie. The other authors report no conflicts.
Footnotes
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.116.013144/-/DC1.
- Received February 15, 2016.
- Revision received March 16, 2016.
- Accepted March 22, 2016.
- © 2016 American Heart Association, Inc.
References
Jump to
Related Internet Resources:
This Issue
Article Tools
Share this Article
- Clinical Scales Do Not Reliably Identify Acute Ischemic Stroke Patients With Large-Artery Occlusion
