Detection of Early CT Signs of >1/3 Middle Cerebral Artery Infarctions
Interrater Reliability and Sensitivity of CT Interpretation by Physicians Involved in Acute Stroke Care
Background and Purpose—This study had 2 goals: (1) to assess interrater reliability of academic neuroradiologists when classifying acute infarction by CT scan as >1/3 middle cerebral artery (MCA) involvement, <1/3 MCA involvement, or no infarction and (2) to determine the sensitivity of physicians potentially involved in acute stroke treatment in detecting >1/3 MCA acute infarctions. Studies of tissue plasminogen activator show an association between early signs of major infarction and poor outcome. The American Academy of Neurology and the American Heart Association recommend avoiding thrombolysis if early signs of major infarction are present.
Methods—We presented 25 scans (normals, acute infarctions, and old infarctions) to 3 academic neuroradiologists. A scoring sheet based on Alteplase Thrombolysis for Acute Noninterventional Therapy in Ischemic Stroke (ATLANTIS)/CT Summit criteria was used to determine >1/3 MCA territory involvement. Nine of the 25 scans were presented again to assess intrarater reliability. We recalculated results of our previous study in which physicians interpreted infarction scans, now designating the scans as >1/3 MCA, <1/3 MCA, or normal, as determined by the neuroradiologists.
Results—All 3 neuroradiologists agreed on no infarction, <1/3 MCA, and >1/3 MCA on 64% of the scans. Neuroradiologist test-retest agreement was 96% for >1/3 MCA territory. Overall sensitivity for emergency physicians, neurologists, and general radiologists for detecting the presence of infarction in scans rated as >1/3 MCA was 78%.
Conclusions—Neuroradiologists can achieve moderate agreement in detecting >1/3 MCA infarction. The emergency physicians, neurologists, and general radiologists tested were reasonably skilled at detecting >1/3 MCA acute infarction. However, their performance did not reliably identify all patients who have early CT infarct signs that place them at increased risk for cerebral hemorrhage after thrombolytic therapy.
In contradistinction to intracranial hemorrhage, which is an undisputed exclusion criterion for thrombolytic therapy, the importance of the presence of early signs of acute infarction on CT is controversial. Several large clinical trials have demonstrated that early signs of major infarction on CT are predictive of a higher risk of symptomatic hemorrhage after thrombolytic administration.1 2 3 4 In the European Cooperative Acute Stroke Study I (ECASS I),5 subjects treated with tissue plasminogen activator (tPA) who exhibited early signs of infarction involving >1/3 middle cerebral artery (MCA) territory had a 3.5-fold increase in parenchymal hemorrhage. The National Institute of Neurological Disorders and Stroke (NINDS) tPA trials found that patients who exhibited early signs of infarction on initial CT were 8 times more likely to experience a symptomatic hemorrhage if they received tPA than those receiving placebo.6 Early signs of major infarction have been used as exclusion criteria in large clinical trials of both intra-arterial and intravenous thrombolytic therapy.2 3 7 8
Results from the ECASS I and the NINDS tPA trials provided the background for national recommendations regarding the use of early CT signs of major infarction as a relative or an absolute exclusion criterion. Food and Drug Administration labeling and the National Stroke Association consensus statement regarding tPA in acute ischemic stroke urges caution in administering tPA to patients with CT signs of early infarction.9 National guidelines from the American Heart Association and the American Academy of Neurology state that thrombolysis should be avoided if early signs of major infarction are identified on CT.10 11 However, these recommendations are not universally agreed on.6 12
For the exclusion criteria of early signs of involvement of >1/3 MCA territory to be useful in clinical practice, physicians must be able to reliably identify scans with this finding. Two published studies and 1 abstract have examined neuroradiologists’ agreement on a common operational definition of major infarction, ie, infarction involving >1/3 of the MCA territory.13 14 15 These studies found 72% to 86% agreement among neuroradiologists with regard to whether <1/3 MCA or >1/3 MCA territory was involved in patients scanned within 6 hours of stroke onset. A recent study has examined the ability of neuroradiologists and other physicians to identify signs of early infarction by using the images from the ECASS I trial, but this study did not restrict their sample to scans exhibiting >1/3 MCA territory involvement.16 The ability of nonneuroradiologists to reliably detect CT findings of >1/3 MCA infarctions has not been characterized.
We previously reported on physicians’ abilities to identify signs of early infarction and hemorrhage.17 In that study, we classified acute infarction scans as easy, intermediate, or difficult to interpret on the basis of our collective judgment of lesion conspicuity. Conspicuity was defined as how hard the lesion was to visualize by making a global assessment of many factors, including the degree of hypodensity and tissue contrast, the amount of sulcal effacement, mass effect, and the size of the lesion. Overall sensitivity for acute infarction scans according to the conspicuity of the lesion was 90% for easy scans, 69% for intermediate scans, and 30% for difficult scans. Because of the growing consensus regarding the importance of recognizing infarction involving >1/3 MCA territory, the present study was performed to reclassify each acute infarction scan as >1/3 MCA, <1/3 MCA, or no infarction and to calculate each subject’s performance on infarcts classified by this scheme. Thus, the present study has 2 components: (1) determination of the interrater agreement of neuroradiologists identifying >1/3 MCA territory infarctions and (2) the sensitivity of nonneuroradiologists for detecting acute infarctions of >1/3 MCA territory on CT.
Materials and Methods
In preparation for the present study and in our previous study, we identified 54 cranial CT scans exhibiting early signs of infarction, hemorrhage, calcification (used as impostors for hemorrhage), old infarction (used as impostors for early infarction), and normals (without calcification).17 We rated each acute infarction scan as easy, intermediate, or difficult to interpret on the basis of our collective assessment of how hard the lesion was to see. Emergency physicians, neurologists, and general radiologists were tested to determine their ability to identify scans exhibiting signs of early infarction regardless of the amount of territory involved.
For the present experiment, we asked 3 academic neuroradiologists to classify scans regarding the presence and size of acute infarction. In individual sessions, each neuroradiologist was presented all 19 acute infarctions, 3 old infarctions, and 3 normal scans from the aforementioned 54-scan library. To facilitate consistent interpretation and optimize performance, we designed a worksheet incorporating Alteplase Thrombolysis for Acute Noninterventional Therapy in Ischemic Stroke (ATLANTIS)/CT Summit criteria8 for involvement of >1/3 MCA territory (Figure 1⇓).1 Although there is no proof that such a worksheet improves the reading of CTs, research indicates that breaking a complex problem into a series of simpler problems improves decision making, as does the provision of structured worksheets.18 The ATLANTIS criteria define >1/3 MCA territory as substantial involvement of ≥2 of the following 4 areas: frontal, parietal, temporal, or both basal ganglia and insula. “Substantial” was not defined in the ATLANTIS/CT Summit criteria and was therefore left up to the neuroradiologist’s discretion. For every scan, each neuroradiologist rated 10 areas (5 regions×2 hemispheres) as substantial, nonsubstantial, or no infarction. Scans were categorized as exhibiting early signs of major infarction if at least 2 of the 3 neuroradiologists agreed that ≥2 areas had substantial involvement. Scans were categorized as normal if at least 2 of the 3 neuroradiologists agreed that there were no signs of acute infarction. All remaining scans were categorized as <1/3 MCA involvement.
To assess intrarater (test-retest) reliability, the same neuroradiologists were shown for a second time 6 acute infarction scans that evoked disagreement during the first round of interpretation and 3 normal scans. Unlike the first round, in this round, the 3 neuroradiologists were told which side of the body was symptomatic before completing the CT interpretation worksheet. Having reclassified each infarction according to the ATLANTIS criteria, we used the results from our original study to recalculate the sensitivity of the emergency physicians, neurologists, and radiologists for detection of >1/3 MCA and <1/3 MCA acute infarctions.
Interrater and test-retest reliability are expressed as percent agreement. We chose not to present κ values because individual raters were not aware of the correct marginal totals and therefore percentage agreement is both simpler and more appropriate. CIs are calculated by using the robust clustered logistic regression procedure in STATA 6.0 (Stata Corp), with the subject used as the clustering variable, to account for the potential nonindependence of the readings of each subject.19
Nine scans were classified by the neuroradiologists as demonstrating >1/3 MCA infarction. The 3 neuroradiologists were unanimous in their classification of 6 of these and also agreed completely on which areas of the brain were involved. For the other 3 scans deemed >1/3 MCA, there was 1 rater who scored the scans as <1/3 MCA (Figure 2⇓). Ten scans were classified as normal, including 6 scans with unanimous agreement. Three of the normal scans were read by the contradicting reviewer as <1/3 MCA, and 1 of the normal scans was read as >1/3 MCA. The conspicuity of the infarction, based on the authors’ collective judgment of the obviousness of the lesion, and the size of the infarction, based on the neuroradiologists’ assessment of the ATLANTIS criteria, were not strongly correlated (Table 1⇓). Five of the 8 scans that had an official reading of acute infarction and had been classified as “difficult” in the original study were classified as “no infarction” by the panel of neuroradiologists. We examined the effect of changing these “difficult” infarctions to “no infarction” on the results presented in our original study and found that no parameter changed by >2%.17 Interrater agreement regarding specific brain regions varied.
Test-retest agreement regarding the presence of >1/3 MCA infarction was 96% (26 of 27, from 3 neuroradiologists reading 9 scans). One neuroradiologist changed the rating of a single scan from <1/3 to >1/3 MCA territory involvement, but this did not change the rating of the scan because it had already been deemed >1/3 by the other raters. For individual regions (frontal, parietal, temporal, insula and basal ganglia), judgments of the magnitude of involvement of acute infarction were the same on retesting for 93% (252 of 270, from 3 neuroradiologists reading 10 individual regions of 9 scans) of the readings. All 18 of these changes occurred on 4 scans. Of the 18 individual regions that changed, 3 (1%) were a decrease in the magnitude of involvement of the region (from substantial to nonsubstantial involvement or nonsubstantial involvement to no infarct), whereas 15 (5%) were an increase in the magnitude of involvement of the region. For the 19 acute infarction scans, unanimous agreement occurred in 47% of parietal, 53% of basal ganglia, 58% of frontal, 63% of temporal, and 89% of insular regions.
Sensitivity for the detection of >1/3 MCA acute infarctions, based on 282 readings of 9 scans, was 78% (CI 71% to 83%). Emergency physicians had a sensitivity of 62% (CI 50% to 72%); neurologists, 89% (CI 80% to 94%); and general radiologists, 86% (CI 77% to 92%) (Table 2⇓). Each physician read between 1 and 4 scans with >1/3 MCA infarction. Sensitivity of the individual physicians ranged from 0% to 100%.
Many rural and smaller hospitals do not have full-time neuroradiologists on staff. Even in larger hospitals, neuroradiologists may be unavailable for urgent CT interpretation within the 3-hour time window for administration of tPA for acute ischemic stroke. In these situations, physicians who are involved in the care of acute-stroke patients, such as emergency physicians, neurologists, and general radiologists, must be adept in interpreting the signs of early infarction to identify appropriate thrombolytic candidates. The present study asks whether physicians in these specialties are able to detect early signs of >1/3 MCA territory infarction on CT scans with a high degree of accuracy.
Overall, physicians achieved a sensitivity of 78% for identifying the signs of early infarction of >1/3 MCA territory. This finding is consistent with a study by Grotta et al,20 which examined NINDS trial investigators, neurologists, emergency physicians, radiologist fellows, and stroke fellows agreement on acute infarctions involving >1/3 MCA territory. This study found a range of agreement from 68% to 85%, with an average of 77% agreement.20 This level of performance suggests that although a majority of the patients at increased risk of hemorrhage from this risk factor would be identified, a significant proportion would not be detected. Improving physicians’ abilities to recognize early signs of major infarction could enhance our ability to provide advice regarding the risk of hemorrhagic complications. A recent study by Wardlaw et al16 also showed that a significant number of acute infarctions are not identified by practitioners involved in the care of stroke patients. In fact, only 45% of patients were correctly identified as exhibiting >1/3 MCA involvement. Although our results identify deficiencies in identifying >1/3 MCA territory involvement, they cannot be directly compared because of different testing methods, different physician groups, and even different severity of ischemic changes exhibited on CT scans.16
The interrater agreement among the 3 neuroradiologists was 64% overall, but it was 86% for identifying >1/3 MCA infarcts. The latter number is comparable to that found in previous studies involving neuroradiologists’ reading of >1/3 MCA. von Kummer et al13 examined the readings of 3 neuroradiologists for >1/3 MCA involvement and found 86% agreement, whereas Marks et al14 also tested the readings of 3 neuroradiologists and found 72% agreement.
It is unclear to what extent the structured form based on ATLANTIS/CT Summit criteria (Figure 1⇑) improved or hindered the uniform interpretation of the scans. Test-retest reliability for neuroradiologists was excellent, with only 1 neuroradiologist changing a reading on a single scan from <1/3 MCA to >1/3 MCA involvement. Even judgments of individual regions exhibited a high degree of stability between readings for each neuroradiologist. Providing the side of the patient’s deficit to this sample of neuroradiologists at the second reading did not change their original interpretation of scans exhibiting >1/3 MCA involvement. However, providing the side of the patient’s deficit resulted in an increase in the detection of minor signs of early infarction (<1/3 MCA involvement). This effect of lateralization information likely reflects both an increase in the attention of the neuroradiologists to the hemisphere in question and the facilitation of interpretation of ambiguous areas.
There was a suggestion that the overall conspicuity of lesions and the amount of MCA territory involved influenced the identification of acute infarction differently among the 3 specialty physician-interpreter groups. Lesion conspicuity most strongly influenced the ratings of emergency physicians and moderately influenced the ratings of neurologists and radiologists. Lesion size modestly affected the ratings of emergency physicians and neurologists but did not affect the ratings of radiologists. We speculate that these differences by physician specialty may reflect differing scan-reading strategies. Interpreters who primarily use a global-gestalt recognition strategy (eg, emergency physicians) would likely be more vulnerable to size influences than interpreters who use a strategy involving serial inspection of individual signs (eg, radiologists).
Our convenience sampling method is the main threat to the external validity of the study. The physicians tested may not be representative of all physicians who treat stroke patients, and it could be argued that they are likely better than the typical reader.17 The scans chosen for the library were representative scans exhibiting early infarctions of differing severity. This selection allowed us to survey physicians’ abilities to detect early signs of infarction with a broad range of findings. Because of the composition of our scan library, we cannot precisely answer how our subjects would perform on scans confined to within 3 hours of symptom onset. However, it would be reasonable to extrapolate that the performance would be similar to the readings of scans rated as “difficult” acute infarction, because these images generally exhibited a minimal amount of tissue hypoattenuation typical of hyperacute CT studies.
Our previous study17 demonstrated that physicians who are involved in providing acute stroke care (ie, emergency physicians, neurologists, and general radiologists) attained a sensitivity of 82% for recognizing hemorrhage on CT. The present study shows that the sensitivity of these physicians for identifying acute infarction involving >1/3 MCA territory is 78%. Both studies support the concept that typical practicing physicians do not achieve near-perfect sensitivity for CT contraindications to thrombolysis. Interpretation of acute CT scans may be improved by providing additional training to physicians involved in acute stroke care,21 including the use of structured worksheets or web-based educational programs, by using teleradiology or other systems that permit immediate scan interpretation by expert readers, or by developing new modalities for selecting patients who will most benefit from thrombolytic agents.
Mary Kalafut, MD, was funded in part by an American Heart Association training grant. David Schriger, MD, was funded in part from an unrestricted grant for health services research from the MedAmerica Corp. The authors wish to thank the physicians who participated in the study, especially J. Bentson, MD, S. El Saden, MD, and P. Villablanca, MD. We also thank Richelle Cooper, MD, for data analysis.
↵1 To address the problems encountered defining “major acute infarction” in previous studies, the investigators in the ATLANTIS trial convened a summit meeting that included Drs Fox, Marks, Patel, and von Kummer. They developed operational definitions for categorizing the size of infarctions. With the consent of Dr Micheal Marks and Genentech (personal communication, August 1998), we developed the form (Figure 1⇑) to facilitate the consistent evaluation of these rules.
- Received December 15, 1999.
- Revision received April 26, 2000.
- Accepted April 26, 2000.
- Copyright © 2000 by American Heart Association
Larrue V, von Kummer R, del Zoppo G, Bluhmki E. Hemorrhagic transformation in acute ischemic stroke: potential contributing factors in the European Cooperative Acute Stroke Study. Stroke. 1997;28:957–960.
The NINDS t-PA Stroke Study Group. Intracerebral hemorrhage after intravenous t-PA therapy for ischemic stroke. Stroke. 1997;28:2109–2118.
National Stroke Association Consensus Statement. Stroke: the first hours: emergency evaluation and treatment guidelines. In: Stroke Clinical Updates. Englewood, Colo: National Stroke Association; 1997:1–14.
Quality Standards Subcommittee of the American Academy of Neurology. Practice advisory: thrombolytic therapy for acute ischemic stroke. Neurology. 1996;47:835–839.
Adams HP, Brott T, Furlan A, Gomez C, Grotta J, Helgason C, Kwiatkowski T, Lyden P, Marler J, Torner J, et al. Guidelines for Thrombolytic Therapy for Acute Stroke: a Supplement to the Guidelines for the Management of Patients with Acute Ischemic Stroke: a statement for healthcare professionals from a Special Writing Group of the Stroke Council, American Heart Association. Stroke. 1996;27:1711–1718.
The NINDS t-PA Stroke Study Group. Generalized efficacy of t-PA for acute stroke: subgroup analysis of the NINDS t-PA Stroke Trial. Stroke. 1997;28:2119–2125.
Marks MP, Holmgren E, Fox A, Patel S, von Kummer R, Froehlich J. Evaluation of early computed tomographic findings in acute ischemic stroke. Stroke. 1999;30:389–392.
Marks MP, Lansberg MG, Howard AI, Albers GW. Comparison of CT and DWI for detection of early infarct signs. Neurology. 1999;52:A453. Abstract.
Wardlaw JM, Dorman PJ, Lewis SC, Sandercock PA. Can stroke physicians and neuroradiologists identify signs of early cerebral infarction on CT? J Neurol Neurosurg Psychiatry. 1999;67:651–653.
de Dombal FT, Dallos V, McAdam WA. Can computer aided teaching packages improve clinical care in patients with acute abdominal pain? BMJ. 1991;302:1495–1497.
Stata Corp. 1999 Stata Statistical Software: Release 6.0. College Station, Tex: Stata Corp; 1999.
Grotta JC, Chiu D, Lu M, et al. Agreement and variability in the interpretation of early CT changes in stroke patients qualifying for intravenous rtPA therapy. Stroke.. 1999;30:1528–1533.
von Kummer R. Effect of training in reading CT scans on patient selection for ECASS II. Neurology. 1998;51:S50–S52.