Inadequacy of Clinical Scoring Systems to Differentiate Stroke Subtypes in Population-Based Studies
Background and Purpose We undertook to examine the usefulness for epidemiological studies of two well-known validated clinical scoring methods, the Guys’ Hospital Stroke score and the Siriraj Hospital Stroke score, to classify strokes into the two main types, hemorrhagic and ischemic, in epidemiological studies.
Methods Patients from a population-based stroke register who received either a CT scan or an autopsy were retrospectively scored using the two clinical scoring methods. The scores were then compared with the CT scan and autopsy results to determine the sensitivity, specificity, and positive predictive value for intracranial hemorrhage (primary intracerebral and subarachnoid hemorrhage) and ischemic stroke.
Results Over a 12-month period, 554 patients from a population-based study underwent CT scanning. Films or autopsy reports were available for 521 patients, and of these, sufficient clinical information to calculate the Guys’ Hospital Stroke score and the Siriraj Hospital Stroke score was available for 464 and 475 patients, respectively. For the Guys’ Hospital Stroke score, the sensitivity and specificity for intracranial hemorrhage were 31% and 95%, respectively; the positive predictive value was 73%. The sensitivity and specificity for ischemic stroke were 78% and 70%, respectively, and the positive predictive value was 86%. For the Siriraj Hospital Stroke score, the sensitivity and the specificity for intracranial hemorrhage were 48% and 85%, respectively; the positive predictive value was 59%. The sensitivity and specificity for ischemic stroke were 61% and 74%, respectively, and the positive predictive value was 84%.
Conclusions This validation study suggests that both clinical scores lack sufficient validity to be used in epidemiological studies for classification of stroke types and should probably not be used in the randomization of patients into treatment trials using thrombolytic or antithrombotic drugs in the absence of diagnostic information based on neuroimaging techniques.
The definition of stroke used for epidemiological studies is based on clinical presentation.1 2 3 4 Since the advent of CT and MRI, greater attention has been focused on changes in the occurrence and case-fatality of the different pathological types of stroke. Although the criteria for the definition of stroke are relatively accurate for determining the presence or absence of stroke, the differentiation of pathological types requires brain imaging.5 6 7 Experience from population-based studies has shown that up to a quarter of all stroke events are treated at home or in community-based long-term care institutions, making the differentiation of stroke types very difficult.8 9 10 Nevertheless, the ability to classify stroke into hemorrhagic and ischemic types in epidemiological research would improve understanding of the nature of stroke and provide clues to its etiology and potential interventions in the acute stage.
To achieve this goal in the absence of diagnostic investigations, clinical scoring protocols have been produced by Allen11 and Poungvarin et al.12 These two scoring protocols are the only ones currently available that have been validated against both postmortem and CT scan results. They are designed to give an objective score based on clinical variables shown to be significantly different for hemorrhagic and ischemic strokes. We tested these two clinical scores in a subset of patients from a large population-based stroke study to examine the feasibility of applying one of the scores to the whole data set to allow differentiation between hemorrhagic and ischemic stroke.
Subjects and Methods
Stroke is defined according to World Health Organization clinical criteria as “a rapidly developing clinical syndrome of focal (or global in the case of subarachnoid hemorrhage) disturbance of cerebral function lasting longer than 24 hours (unless interrupted by surgery or death), presumably of vascular origin.”1 The definition includes patients with subarachnoid hemorrhage. It does not include subdural or extradural hemorrhage, brain tumor, trauma, or transient ischemic attack.
All residents of the Auckland region (total population 945 000, 1991 census) who had a stroke in the 12-month period ending March 1, 1992, were eligible for inclusion in the study. The details are published elsewhere13 ; briefly, the study was characterized by multiple and overlapping case-finding sources including efforts to detect persons with acute stroke events not admitted to a hospital. As soon as the event came to the attention of the study team, the clinical signs were checked against the criteria for inclusion. Patients were interviewed by trained nurses in association with a neurologist (N.E.A.). A structured personal interview with a precoded questionnaire was administered covering the onset of the stroke and other medical and social factors. If the patient was deceased or unable to communicate, these questions were directed to a close relative or friend. Details from hospital notes were abstracted 1 month after the stroke event or at discharge, whichever came first. Patients who gave a history of stroke were classified as having a history of stroke only after careful review of the available medical information. A recurrent event during the study period was considered to have occurred if more than 28 days had elapsed from the initial episode that brought the patient into the study.
During the study period, Auckland was served by three CT scanners in two of the four main public hospitals. Two private CT scanners were used by a few patients registered in the study. No nuclear MRI was available at the time of the study.
Details of the timing and results of the CT scans were collected using a structured form. If multiple lesions were seen on the CT scan, the one most consistent with the clinical picture was used for diagnostic purposes. Only CT scans taken within 15 days of the ictus were included in the analysis to eliminate the possible misdiagnosis of resolving intracerebral hemorrhage as ischemic stroke.14
Differentiation between hemorrhagic and ischemic strokes was also possible from autopsy examinations undertaken on some of the patients enrolled in the study. The majority of the autopsies were performed at the coroner’s request. The diagnosis was obtained from the pathologist’s report, and if multiple lesions of varying ages were seen, the most recent lesion (or the one stated to have caused death) was used. If both CT scan and autopsy reports were available, the CT scan was used because in all cases that information was more complete.
Clinical details required for the Guys’ Hospital Stroke (GHS) score (also known as the Allen score)11 and the Siriraj Hospital Stroke (SHS) score12 were obtained from the study interview and from the hospital notes (see “Appendix”). Both scores were calculated by a computer algorithm from this data.
In the absence of clear definitions in the original scales, some assumptions were made. When the value for diastolic blood pressure at 24 hours after admission was not available, the nearest recording at less than 24 hours was used. Patients were assumed to be fully conscious if they had a value of greater than 13 in the Glasgow Coma Scale (GCS)15 or if it was reported that the patient had not lost consciousness within the first 2 hours (for the SHS score) or at 24 hours (for the GHS score). Patients were defined as drowsy if they had a GCS score of 8 to 13 inclusive, and patients with GCS scores of 7 or less were assumed to have been unconscious unless it was reported that the patient had lost consciousness in the specified time period. A history of hypertension was determined at the initial interview as well as from the hospital notes.
Missing variables in the calculations were treated in one of two ways. If they did not affect the overall score (eg, if a positive variable was missing but the score was already greater than +24 in the GHS score or greater than +1 in the SHS score), then the missing values were adjusted to zero and the classification stood. If the value could have changed the person’s diagnostic category, then the overall score was left as missing data.
Statistics were computed on a personal computer using sas version 6.08 statistical software. Relative risks and age-stratified analyses controlling for age, sex, and ethnicity were calculated using the Mantel-Haenszel method within sas. Sensitivity, specificity, and positive predictive value were calculated using standard formulas.16
Overall, 1803 stroke events in 1761 persons occurred in the 12 months up to March 1, 1992, of which 1305 (72%) were first-ever events. Of the total events, 1308 (73%) persons were admitted to a public hospital as inpatients; 469 (26%) were diagnosed and treated in the community (either at home or in a long-term private care institution), and the remainder (1%) died before receiving any treatment. A wide range of ages was found (19 to 101 years), with a median age for all patients of 74 years (men, 70 years; women, 77 years); 24% were under 65 years of age and 48% were over 75 years.
Of all events registered, 554 (31%) patients had a CT scan; of these, CT information was collected for 529. The majority of the scans (69%) were performed at Auckland Hospital. Most scans were performed on hospital inpatients; only 11 patients not admitted to a hospital were referred for a CT scan. The charts of those admitted to a hospital show that 42% received a CT scan.
There were 484 (87%) CT scans undertaken within 15 days of the ictus. Of the eligible CT scans, the median delay from onset to scan was 1 day (range, 0 to 15 days), with 90% being performed within 8 days. There was a significant age difference between those patients receiving a scan and those not receiving one. Patients of younger age were much more likely to be selected for a CT scan: one half of all patients receiving scans were less than 65 years of age, and only 28% were 75 years old or greater (P<.005). Patients with subarachnoid hemorrhage were also significantly more likely to receive a CT scan (odds ratio, 1.66; 95% confidence interval, 1.40 to 1.97) after controlling for age. However, in an age-stratified analysis, no significant variation in ethnicity or sex predicted whether the person received a CT scan.
Of the 51 autopsies undertaken, 13 were disregarded in the analysis because CT information was available for the patient, and one patient was excluded because of the unclear report. The data from 484 CT scans and 37 autopsies were combined, giving a total of 521 events with a confirmed diagnosis. Subsequent analyses were based on those with a definitive diagnosis, that is, those from the combined set of CT scans undertaken within 15 days and autopsy reports.
Of the 521 patients who received a diagnostic CT scan or autopsy, information was available to calculate the GHS score for 472 (91%) and for 485 (93%) for the SHS score. Table 1⇓ summarizes the sources of definitive diagnoses.
Applying the recommended optimum cutoff points for each scale, the cases with definitive diagnoses were classified by the GHS and SHS scales as probable hemorrhagic strokes (10% and 14%, respectively) or probable ischemic strokes (54% and 43%, respectively). The remainder were classified as uncertain. In comparison, those who had no CT or autopsy and therefore no definitive diagnosis were scored as hemorrhagic in 5% and 10%, respectively, and as ischemic in 80% and 71% of cases, respectively. On a χ2 test for homogeneity, the differences between the two groups were highly significant (P<.0005) for both scales, demonstrating that definitive diagnoses were available only for selected events.
With CT scans and autopsy findings, the 521 events were classified into one of three diagnostic groups independent of clinical observations or other test results. Intracranial hemorrhages (ICH) included 82 subarachnoid hemorrhages (16%), 97 intracerebral hemorrhages (19%), and 342 ischemic strokes (66%). These diagnoses were each compared with the calculated GHS score and the SHS score.
The results for the GHS score are shown in Tables 2⇓ and 3⇓. The GHS score had a sensitivity (probability of a positive test in people with the disease) of 31% and a specificity (probability of a negative result in people without the disease) of 95% for ICH. The positive predictive value (probability of the person having the disease when the test is positive) for ICH was 73% (Table 2⇓). For ischemic stroke, the GHS score had a sensitivity of 78%, a specificity of 70%, and a positive predictive value of 86% (Table 3⇓).
The results for the SHS score are shown in Tables 4⇓ and 5⇓. For ICH, the SHS score had a sensitivity of 48%, a specificity of 85%, and a positive predictive value of 59% (Table 4⇓). For ischemic stroke, the SHS score had a sensitivity of 61%, a specificity of 74%, and a positive predictive value of 84% (Table 5⇓).
The findings suggest that these two commonly used, validated clinical scoring systems11 12 do not have sufficient sensitivity to allow classification of stroke into the two main types, hemorrhagic or ischemic, in epidemiological studies of stroke in which the score is determined retrospectively. The SHS score was more sensitive for hemorrhagic stroke than the GHS score, but both were unacceptable. We also have concerns about their use as a diagnostic screening procedure for either clinical trials or decisions about treatment in ordinary clinical practice.
Our results are in conflict with previously reported studies that claim sensitivities greater than 75% for detecting hemorrhage.11 12 17 18 19 Another study supported these scores, although the sensitivities for the detection of hemorrhage were 38% and 61% for the GHS and SHS scores, respectively.20 However, our findings do support a recent work that highlights limitations of these scoring systems in clinical trials.21 The latter study focused on patients admitted to an acute stroke unit, unlike our study, which included patients admitted to acute medical and neurological wards as well as geriatric wards. Some of the differences between our study and the others relate to whether the scoring systems are for epidemiological classification or for clinical trials in which the intention is to eliminate hemorrhagic strokes. In addition, some studies have unjustifiably excluded patients with an indeterminate score from the calculations for specificity and sensitivity, effectively increasing both measures.18 20 Another study used a “best” cutoff point to give the highest sensitivity at minimum cost to specificity.17
Another explanation for the differences among the studies relates to the selection of patients. Although our large population-based study is characterized by its completeness, the patients who received a CT scan in our study were younger and more likely to have been treated in the hospital. This may have been due in part to limited availability of CT scanners within the population area. Very few of the patients treated out of the hospital received a CT scan. It is therefore reasonable to compare our series with the other hospital-based series.
The original GHS score was based on consecutive hospital patients, most of whom received a CT scan.11 Patients with severe symptoms leading to early death would not have been considered in the original classification. In our study, all patients shown to have an ischemic stroke by CT scan or autopsy, but who were predicted by clinical score to have a hemorrhage, had more severe symptoms and therefore high “apoplectic onset” scores. Whereas some studies have omitted them, our analysis included the patients with severe cases who were admitted to the hospital but subsequently died within 24 hours and were autopsied.
Patient selection would have been influenced by age. The GHS score was developed with patients under the age of 76 years, yet nearly half the people who had strokes in our study were over this age. The restriction to the younger population in the development of the GHS score is likely to increase the proportion of ICH compared with ischemic stroke, since ICH (especially subarachnoid hemorrhage) is more prevalent in the younger population.4 19 However, when we restricted the analysis to those under 76 years of age, sensitivities and specificities were similar, moving up or down by less than 4%. A similar problem of patient selection has been noted with the SHS score. This scoring system was developed in Thailand; a higher prevalence of uncontrolled hypertension in their population may have introduced an “ethnic” bias reflected in the higher rate of ICH in their study population.12
Both scores lack formal definitions for some variables, and this is rarely addressed. The main problem relates to the “level of consciousness” because it is such an important weighting factor in both scoring protocols. We attempted to make the definition for the level of consciousness more objective by substituting the GCS.15 This may partly explain the lower sensitivities because the use of the GCS would tend to underscore patients with high GCS scores and mild drowsiness. A consistent measure of level of consciousness is essential in large population-based studies when reliance on a single physician is not practical.
It is likely that, when resources are limited, CT scanning is more often undertaken to assist the diagnosis when the clinical presentation is atypical or hemorrhage is suspected. Clinicians may not request a CT scan when the clinical signs are strongly suggestive of an ischemic stroke.22 In our study, the events in patients who received a CT scan were not typical of all events that occurred in the 12-month period.
Although the diagnostic judgments of the neuroradiologists and pathologists in our study were not validated, studies have shown no real difference between radiologists in the differentiation between hemorrhagic and ischemic strokes23 ; furthermore, most (78%) of our CT scans were reported either by a single neuroradiologist or in conjunction with a neurologist (N.E.A.).
The validity of scans performed more than 15 days from the onset of the stroke has also been questioned.14 In only two of the studies were patients excluded if the CT scan was obtained more than 15 days after the onset of the stroke.12 18 For the 521 events included in our analysis, median delay to scan was 1 day and the mean was 2.8 days. For the GHS score, the time delay was not considered as a factor, although the mean time to scan was 8.4 days (range, 1 to 21 days). It is therefore probable that several hemorrhages were incorrectly diagnosed as ischemic strokes.14 24 This could have led to a score that erroneously included small hemorrhages as ischemic strokes. Our data show that people with hemorrhage who had a low GHS score all scored zero for the “apoplectic onset.” This also occurred in the Siriraj Hospital study despite subjects receiving early CT scans.
In general, both scoring methods tend to classify severe strokes as hemorrhagic and strokes of less severity as ischemic regardless of the etiology involved. If all strokes are regarded as ischemic strokes unless shown otherwise, in populations largely of Caucasian descent the implications of the inaccuracy occurring with the clinical scoring techniques may be minimal due to the relatively high proportion of ischemic strokes among strokes generally.2 3 4 5 11 12 17 18 19 25 This is not the case for populations in which the proportion with hemorrhagic stroke is higher, eg, in Japan and China.26 27
Not only did we find the clinical scores inadequate for our epidemiological purposes, but we believe that the use of these classification systems for differentiating stroke type is unreliable in clinical practice; a person treated with antithrombotic or thrombolytic therapy on the basis of these clinical scores is at risk if the stroke is hemorrhagic. Classification errors work both ways: they may prevent the use of a useful therapy for a particular type or they may allow potentially lethal treatments to proceed.
On the other hand, since neuroimaging techniques are unlikely ever to become universally available, there is a need for a clinical scoring method that is able to predict accurately one of the types of stroke. We expect that there will always be a large group of patients whose strokes are not distinguishable by clinical signs and symptoms alone. It appears that the classification of stroke type will remain beyond the capacity of large community-based epidemiological studies in many parts of the world.
Clinical Construction of Scoring Systems
The scores are derived from relating the clinical features to the scores shown, which are summed. In both scales, a high positive score suggests hemorrhagic stroke and a negative score, ischemic stroke.
Guys’ Hospital Stroke Score
Apoplectic onset (loss of consciousness, headache within 2 hours, vomiting, neck stiffness): one or none of these, 0; two or more, +21.9.
Level of consciousness (24 hours after admission): alert, 0; drowsy, +7.3; unconscious, +14.6.
Plantar responses: both flexor or single extensor, 0; both extensor, +7.1.
Diastolic blood pressure (24 hours after admission, in mm Hg): ×+0.17.
Atheroma markers (angina, claudication, diabetes history): none, 0; one or more, −3.7.
History of hypertension: not present, 0; present, −4.1.
Previous event (transient ischemic attack or stroke): none, 0; any number of events, −6.7.
Heart disease: none, 0; aortic or mitral murmur, −4.3; cardiac failure, −4.3; cardiomyopathy, −4.3; atrial fibrillation, −4.3; cardiomegaly (from x-ray), −4.3; myocardial infarct (within 6 months), −4.3.
Siriraj Hospital Stroke Score
Consciousness: alert, 0; drowsy or stupor, 2.5; coma or semicoma, 5.
Vomiting: no, 0; yes, 2.
Headache (within 2 hours): no, 0; yes, 2.
Diastolic blood pressure (in mm Hg): ×+0.1.
Atheroma markers (history of diabetes, intermittent claudication or angina): none, 0; one or more, −3.
Thresholds of Validity
The cutoff level for the scores has been validated in both the Guys’ Hospital Stroke score11 and the Siriraj Hospital Stroke score12 at 90% certainty for one or other classification. The values that give this degree of certainty are as follows.
Guys’ Hospital Stroke Score: hemorrhage (>90% certainty), score greater than +24; ischemic stroke (>90% certainty), score less than +4; and uncertain, score of +4 to +24.
Siriraj Hospital Stroke Score: hemorrhage (>89% certainty), score greater than +1; ischemic stroke (>93% certainty), score less than −1; and uncertain, score of −1 to +1.
We thank the Health Research Council of New Zealand and the National Heart Foundation of New Zealand for their support. We would also like to thank Dr Ayton Hope for his work in interpreting the CT scans, and Dr Richard Lindley, Dr Craig Anderson, Dr Alexandre Kalache, and Professor Geoffrey Donnan for valuable comments made during the preparation of this manuscript. We would also like to thank Wendy Smidt for her valuable secretarial assistance. Dr Bonita is the Masonic Senior Research Fellow and Joanna Broad is the Masonic Research Analyst in the University Section of Geriatric Medicine.
- Received April 11, 1995.
- Revision received May 23, 1995.
- Accepted May 23, 1995.
- Copyright © 1995 by American Heart Association
The WHO MONICA Project Principal Investigators. The World Health Organization MONICA project (monitoring trends and determinants in cardiovascular disease): a major international collaboration. J Clin Epidemiol. 1988;41:171-184.
Harrison MJG. Clinical distinction of cerebral hemorrhage and cerebral infarction. Postgrad J Med. 1980;56:629-632.
von Arbin M, Britton M, deFaire U, Helmers C, Miah K, Murray V. Accuracy of bedside diagnosis of stroke. Stroke. 1981;12:288-293.
Bonita R, Beaglehole R, North JDK. Event, incidence and case fatality rates of cerebrovascular disease in Auckland, New Zealand. Am J Epidemiol. 1984;120:236-243.
Asplund K, Bonita R, Kuulasmaa K, Rajakangas A-M, Feigin V, Schaedlich H, Suzuki K, Thorvaldsen P, Tuomilehto J, for the WHO MONICA Project. Multinational comparisons of stroke epidemiology: evaluation of case ascertainment in the WHO MONICA Stroke Study. Stroke. 1995;26:355-360.
Allen CMC. Clinical diagnosis of the acute stroke syndrome. Q J Med. 1983;52:515-523.
Poungvarin N, Viriyavejakul A, Komontri C. Siriraj stroke score and validation study to distinguish supratentorial intracerebral hemorrhage from infarction. Br Med J. 1991;302:1565-1567.
Bonita R, Broad JB, Anderson NE, Beaglehole R. Approaches to the problems of measuring the incidence of stroke: the Auckland Stroke Study, 1991-1992. Int J Epidemiol. 1995;24:535-542.
Dennis MS, Bamford JM, Molyneux AJ, Warlow CP. Rapid resolution of signs of primary intracerebral hemorrhage in computed tomograms of the brain. Br Med J. 1987;295:379-381.
Beaglehole R, Bonita R, Kjellström T. Basic Epidemiology. Geneva, Switzerland: World Health Organization; 1993.
Sandercock PAG, Allen CMC, Corston RN, Harrison MJG, Warlow CP. Clinical diagnosis of intracranial hemorrhage using Guys’ hospital score. Br Med J. 1985;291:1675-1677. Correction in Br Med J. 1986;292:173.
Celani MG, Righetti E, Migliacci R, Zampolini M, Antoniutti L, Grandi FC, Ricci S. Comparability and validity of two clinical scores in the early diagnosis of stroke. Br Med J. 1994;308:1674-1676.
Franke CL, van Swieten JC, van Gijn J. Residual lesions on computed tomography after intracerebral hemorrhage. Stroke. 1991;22:1530-1533.
Sotaniemi KA, Pyhtinen J, Myllylä VV. Correlation of clinical and computed tomographic findings in stroke patients. Stroke. 1990;21:1562-1566.
Ueda K, Hasuo Y, Kiyohara Y, Wada J, Kawano H, Kato I, Fujii I, Yanai T, Omae T, Fujishima M. Intracerebral hemorrhage in the Japanese community, Hisayama: incidence, change in pattern during long-term follow-up, and related factors. Stroke. 1988;19:48-52.