Clinical Prediction With the ABCD2 Score: At What Cost?
To the Editor:
We read with interest the systematic review of the validity of the ACBD system by Giles et al,1 a clinically important topic. Pooled validation is an important step toward establishing the use of a clinical prediction rule. Given its potential to influence organization of services and acute management protocols for transient ischemic attack, this analysis deserves careful consideration.
The authors conclude that the pooled area under the curve (AUC) of 0.72 is indicative of “good predictive value.” First, the AUC is only a global pretest measure of test performance and not a measure of posttest predictive value that can be applied to an individual patient.2 Moreover, the interpretation of this result is that, in randomly selected transient ischemic attack patient pairs with and without the outcome (stroke/no stroke), the ABCD scores may incorrectly discriminate between high and low risk 28% of the time (based on CIs, this may range from 20% to 37% of the time).2 This is a rather wide margin for error in stratifying risk of early stroke after transient ischemic attack and we question whether this is acceptable for clinical decision-making in daily practice. This may mean that a patient with transient ischemic attack may stand a reasonable chance of either being denied rapid investigation (such as carotid ultrasound) or being unnecessarily admitted to the hospital if one depended solely on the ABCD system. The former could result in a missed opportunity to intervene and prevent stroke, whereas the latter could pose a problem when hospital beds are a scant resource. Although the authors state that the ABCD system should not be a substitute for clinical judgment, they do not deal fully with the health costs of such misclassification, a discussion that may greatly benefit specialists and nonspecialists in daily clinical practice. Given the potential for misclassification, can the authors comment on whether it is acceptable to use the ABCD system as a sole first-step prediction rule to guide management in an individual “usual” patient with transient ischemic attack?
Could the authors also please clarify the following and their potential effect on pooled AUC estimates: (1) It is reported that conference abstracts were searched, but the results of a study by Chandratheva et al3 reporting that the ABCD2 score was unrelated to the risk of stroke appears not to have been included; and (2) data from the study by Asimos et al4 appear to refer only to those with disabling stroke (AUC, 0.71; 95% CI, 0.68 to 0.73) and not all strokes (the primary outcome for pooled analysis) for which a much lower AUC (0.59; 95% CI, 0.56 to 0.62) was reported.
The authors report a similar pooled AUC for stroke at 7 days of 0.72 (0.66 to 0.78) and 0.72 (0.63 to 0.82) for the ABCD and ABCD2 scores, respectively. This indicates equivalence of performance, albeit with slightly wider CI for the ABCD2 score. Do the authors have a preference for 1 score over the other?
We would greatly appreciate the authors’ consideration of our questions to help us to fully evaluate the data and make a considered judgment of the usefulness of the ABCD system in day-to-day clinical practice.
Giles MF, Rothwell PM. Systematic review and pooled analysis of published and unpublished validations of the ABCD and ABCD2 transient ischemic attack risk scores. Stroke. 41: 667–673.
Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993; 39: 561–577.
Asimos AW, Johnson AM, Rosamond WD, Price MF, Rose KM, Catellier D, Murphy CV, Singh S, Tegeler CH, Felix A. A multicenter evaluation of the ABCD(2) score’s accuracy for predicting early ischemic stroke in admitted patients with transient ischemic attack. Ann Emerg Med. 2010; 55: 201–210. e5.