Measuring Carotid Stenosis on Contrast-Enhanced Magnetic Resonance Angiography
Diagnostic Performance and Reproducibility of 3 Different Methods
Background and Purpose— The aim of this study was to compare diagnostic performance and reproducibility of 3 different methods of quantifying stenosis on contrast-enhanced magnetic resonance angiography (CEMRA), with intra-arterial digital subtraction angiography (DSA) as the reference standard.
Methods— 167 symptomatic patients scheduled for DSA, after screening Doppler ultrasound, were prospectively recruited to undergo CEMRA. Severity of stenosis was measured according to the North American Symptomatic Trial Collaborators (NASCET), European Carotid Surgery Trial (ECST), and the common carotid (CC) methods. Measurements for each method were made for 284 vessels (142 included patients) on both CEMRA and DSA in a blinded and randomized manner by 3 independent attending neuroradiologists.
Results— Significant differences in prevalence of severe stenosis were seen with the 3 methods on both DSA and CEMRA, with ECST yielding the least and NASCET the most cases of severe stenosis. Overall, all 3 methods performed similarly well in terms of intermodality correlation and agreement. No significant differences in interobserver agreement were found on either modality. With CEMRA, however, we found a significantly lower sensitivity for detection of severe stenosis with ECST (79.8%) compared with NASCET (93.0%), with DSA as reference standard.
Conclusions— Uniformity of carotid stenosis measurement methods is desirable because patient management may otherwise differ substantially. All 3 methods are adequate for use with DSA. With CEMRA, however, this study supports use of the NASCET method because of improved sensitivity for detecting severe stenosis.
In clinical practice worldwide, several established measurement methods are used to quantify the severity of internal carotid artery (ICA) stenosis before revascularization. The North American Symptomatic Carotid Endarterectomy Trial (NASCET) method, which uses the normal distal ICA as denominator, is most widely used in North America, whereas the European Carotid Surgery Trial (ECST) method, which uses the estimated width of the original artery at the site of maximal narrowing as denominator, tends to be frequently used in Europe.1–3 Moreover, some centers use the common carotid (CC) method, which uses the proximal common carotid artery (CCA) as denominator because it has previously been shown to be more reproducible.4,5 All these methods have similar prognostic value, and as such, are acceptable means of risk stratification.4
Despite some initial confusion generated by these different methods, their equivalence as well as differences are now generally well-understood.6 For NASCET, the pooled analysis of individual patient data from NASCET, ECST, and Veteran Affairs Trial has confirmed that surgery is associated with a significant risk reduction in severe (70% to 99%) stenosis and a more marginal risk reduction in moderate stenosis (50% to 69%).3 These cutoff points of 50% and 70% stenosis with NASCET method have been shown to be equivalent to 65% and 82%, respectively, for both ECST and CC methods.3,6 According to these guidelines, centers using ECST or CC methods should therefore recommend surgery for severe stenosis (82% to 99%), with more individual risk assessment required for the moderate stenosis group (65% to 81%).3,6
Concerns regarding the 1% to 2% risk of neurological complications associated with DSA have generated a strong trend toward the use of noninvasive modalities, with contrast-enhanced magnetic resonance angiography (CEMRA) emerging as the modality of choice.7,8 Several well-designed studies have found acceptably low misclassification rates, supporting the safe use of CEMRA compared with DSA.9–11 All these previous studies have, however, used the NASCET method for quantifying stenosis. The ECST and CC methods have therefore not been previously validated for use with CEMRA. Although DSA is a purely anatomical technique based on intra-arterial injection, CEMRA is a completely different modality, with different spatial resolution, which relies on physiological blood flow and intravenous injection of contrast medium.7 This may have an influence on the reader’s perception and measurement accuracy made at different locations in the carotid arterial tree. The main objective of this study was, therefore, to directly compare the NASCET, ECST, and CC measurement methods on CEMRA, with DSA as the reference standard.
Materials and Methods
Between August 2000 and January 2003, consecutive symptomatic patients suspected of having ICA stenosis on the basis of screening Doppler ultrasound showing at least 50% stenosis in 1 or more arteries and who were scheduled for DSA were prospectively recruited at a single academic institution to undergo CEMRA. Patients with contraindications for magnetic resonance imaging (MRI) such as claustrophobia or metal were excluded. The study was approved by the Local Ethics Research Committee and all patients gave informed written consent.
Full details of the imaging protocol have been described elsewhere.9 Briefly, DSA was performed by an attending neuroradiologist, with bilateral selective CCA catheterization, with images of each bifurcation acquired on 4 projections (antero-posterior, lateral, and bilateral 45° obliques) with a resolution of 0.32×0.32 mm.
CEMRA was performed on a 1.5-T machine (Signa CV/I; GE Medical Systems). A bolus-timed 3-dimensional breath-hold fast spoiled gradient echo acquisition with an elliptic-centric phase-encode ordering scheme was performed. Zero-filling interpolation was used, resulting in a final interpolated voxel size of 0.45×0.45×0.40 mm. Each carotid bifurcation was recorded onto film as 4 maximum intensity projections identical to the ones used for DSA.
Three experienced neuroradiologists, blinded to clinical history and other diagnostic tests, independently reviewed each MRA and conventional angiogram. Patients’ identifiers on films were masked. The DSA images were arranged in a randomized order and reported over a period of 4 months. The CEMRA images were then subsequently reported over the following 4 months, after re-randomization.
For each study, the projection that demonstrated the tightest narrowing was selected. Four measurements were then made: (1) luminal diameter of the normal distal ICA beyond the bulb where the artery wall becomes parallel; (2) luminal diameter at the site of maximal narrowing; (3) luminal diameter of the estimated original width of the artery at the site of maximal narrowing; and, finally, (4) luminal diameter of the proximal disease-free CCA where the artery wall becomes parallel. The calculations of percentage stenosis according to NASCET, ECST, and CC methods are illustrated in Figure 1.
If no stenosis was present, the degree of stenosis was arbitrarily assigned as 0% for all 3 measurement methods. When an ICA demonstrated a near occlusion with collapse of the distal lumen, degree of stenosis was assigned as 95% for all methods.3 For CEMRA, the presence of a signal void was assumed to represent a severe stenosis for each measurement method.12
For the NASCET method, stenosis was classified as mild (0% to 49%), moderate (50% to 69%), severe (70% to 99%), or complete occlusion. Analysis of 1001 angiograms by Rothwell et al also showed an almost equivalent relationship between the ECST and CC methods.6 Therefore, for both ECST and CC methods, stenosis was classified as mild (0% to 64%), moderate (65% to 81%), severe (82% to 99%), or complete occlusion.
All data were entered in SPSS for Windows (V10.1; SPSS Inc) for analysis. For each measurement method, DSA was used as the reference standard. The final reading used was defined as the mean of the 3 independent readers for all 3 methods. Correlation between CEMRA and DSA was calculated using Pearson correlation coefficient and intraclass coefficient. Agreement between the 2 modalities was assessed using the Bland–Altman method and Cohen Kappa statistics. Sensitivity, specificity, and predictive values were calculated to characterize the ability of CEMRA to detect DSA-defined severe stenosis. Finally, interobserver agreement was determined for each measurement method using Kappa statistics.
One hundred sixty-seven consecutive symptomatic patients (45 women; mean age, 70; range, 41 to 89 years old) were recruited. However, 25 patients were excluded from the final analysis because they failed to complete one of the procedures. Full details of patient characteristics, reasons for exclusion, and image quality have been published elsewhere.9 One hundred forty-two patients, who completed both modalities, remained, yielding 284 arteries for analysis.
Distribution of Disease
Table 1 shows agreement between CEMRA and DSA according to different stenosis categories for the NASCET, ECST, and CC methods, respectively. With DSA, use of NASCET, ECST, and CC methods would have resulted in 71 cases (NASCET), 57 cases (ECST), and 65 cases (CC) of severe stenosis respectively. With CEMRA, use of NASCET, ECST, and CC methods would have resulted in 91 cases (NASCET), 66 cases (ECST), and 81 cases (CC) of severe stenosis, respectively.
There was excellent and almost identical correlation between CEMRA and DSA irrespective of measurement method (Figure 2). Pearson correlation coefficient was 0.956 (P<0.001) for NASCET, 0.954 (P<0.001) for ECST, and 0.956 (P<0.001) for CC methods. Similarly, intraclass coefficient was 0.978 (95% CI, 0.971, 0.983) for NASCET, 0.976 (95% CI, 0.970, 0.981) for ECST, and 0.977 (95% CI, 0.971, 0.982) for CC methods.
Bland–Altman plots between CEMRA and DSA for each measurement method are shown in Figure 3. With each method, there was a slight tendency for CEMRA to overestimate stenosis compared with DSA, with mean biases ranging from 0.6% to 2.4%. The limits of agreement, within which 95% of the differences between the 2 modalities are expected to lie, were very similar for all 3 methods, ranging from ±20.0% to ±20.3% around the mean biases. Categorical agreement for NASCET (kappa=0.73; 95% CI, 0.67, 0.79), CC (kappa=0.72; 95% CI, 0.65, 0.78), and ECST (kappa=0.70; 95% CI, 0.63, 0.77) methods were also similar in magnitude.
Diagnostic performance of CEMRA for identifying DSA-defined severe stenosis for each measurement method is summarized in Table 2. For specificity, positive predictive values, and negative predictive values, there were no significant differences between the 3 methods. However, the use of ECST method on CEMRA was associated with a much lower sensitivity (78.9%) for detection of severe stenosis compared with NACET (93.0%), a difference that was statistically significant (P=0.015, Fischer exact test). The CC method had an intermediate sensitivity of 87.7%.
For both DSA and CEMRA, there was no significant difference among observers for the different measurement methods. For DSA, mean kappa values for different pairs of readers was 0.84 (0.83, 0.84, and 0.86) for NASCET, 0.83 (0.81, 0.84, and 0.85) for ECST, and 0.84 (0.83, 0.84, and 0.85) for CC methods, respectively. For CEMRA, mean kappa values for different pairs of readers was 0.84 (0.83, 0.83, and 0.85) for NASCET, 0.80 (0.78, 0.79, and 0.82) for ECST, and 0.79 (0.77, 0.79, and 0.82) for CC methods, respectively. These differences for CEMRA were not statistically significant (χ2 test, NASCET versus ECST, P=0.298, NASCET versus CC, P=0.198).
It is well-understood that both the ECST and CC methods result in a higher percentage stenosis compared with NASCET, given that the denominator used in NASCET, the distal ICA, is generally smaller than the proximal ICA or CCA.6 To account for this, Rothwell et al demonstrated the mathematical equivalence of the 3 methods through analysis of a large number of angiograms and mathematical regression techniques.3,6 We were therefore surprised that, despite this equivalence, all 3 methods still resulted in significant differences in the prevalence of severe stenosis. For instance, with DSA, only 57 cases would have been classified as severe with the ECST method, compared with 65 severe cases with the CC method and 71 severe cases with the NASCET method. This tendency of the ECST method to “under-report” severity of stenosis compared with NASCET was even more pronounced with the use of CEMRA. Thus, as many as 25 cases (66 severe cases ECST versus 91 severe cases NASCET) would have been potentially managed differently on CEMRA; a statistically significant difference (P<0.005, χ2 test). Our study therefore highlights that while mathematical conversions are useful as an approximate guide, in practice, significant errors of categorization are likely, especially taking into account inherent interobserver variability in measuring carotid stenosis. Rothwell et al reported a rate of misclassification of up to 15% in their original series.6
Unfortunately, in the absence of a gold standard method, it is practically impossible to determine which measurement method is most anatomically correct. Although planimetric measurements of stenosis on excised surgical specimens might theoretically represent a reference standard against which to validate angiographic measurement methods, problems such as surgical damage or specimen shrinkage may limit accuracy.13
The reproducibility of the 3 methods on DSA has been previously studied by several authors. Rothwell et al found a consistently greater reproducibility for the CC method compared with both NASCET and ECST.4 This has been the basis for some centers choosing to use the CC method.5 Vanninen et al found a marginally higher interobserver agreement for the ECST method compared with NASCET.14 In our study, on both DSA and CEMRA, there were no differences between the 3 methods which all had excellent reproducibility, a finding consistent with other studies.14,15
With regards to intermodality correlation and agreement, all 3 methods overall performed similarly. Moreover, using the Bland–Altman method we found similar limits of agreement of ≈±20% between CEMRA and DSA. This suggests that, irrespective of measurement method, sizable disagreements may occur between CEMRA and DSA, findings consistent with previous literature.16,17 However, a substantial part of this disagreement is actually because of the fact that the measurement methods themselves are prone to significant variability, even with experienced readers.9 CEMRA tended to overestimate severity of stenosis; this was marginally more evident with NASCET, as evidenced by the mean biases shown in Figure 3. As illustrated by the spread of the disagreements (Figure 3), differences between CEMRA and DSA were more pronounced with milder degrees of stenosis but this observation was consistent across all 3 measurement methods.
This is the first study to our knowledge to explore whether diagnostic performance of CEMRA varies according to the measurement method used. All 3 methods also had very similar performance characteristics with regard to specificity and predictive values. However, a finding of clinical and statistical significance was that ECST was associated with a much lower sensitivity (78.9%) compared with both NASCET (93.0%) and CC (87.7%). This may be related to the difficulty of visually estimating the bulbous nature of the unseen normal proximal ICA on CEMRA, with its intrinsically lower spatial resolution. Despite the fact that highly impressive submillimeter resolution can now be achieved with CEMRA, the true in-plane resolution (0.80×0.90 mm) is still 2- to 3-times less than that of DSA (0.32×0.32 mm).
A limitation of our study is that our readers, despite being experienced neuroradiologists working in a tertiary referral center with a large annual volume of carotid imaging studies, primarily use the NASCET method in routine clinical practice. However, we carefully researched and minimized potential sources of bias before making our measurements, and this is reflected by our high measures of reproducibility for all 3 methods. Results from other groups are, however, needed to confirm the generalizability of our results. Moreover, we used the artery rather than the patient as observational unit. The assumption of independence of each unit is, however, statistically valid because each artery was reported independently from the contralateral side and that stenosis severity between arteries from the same patient is not correlated.
Rothwell et al, in their pooled analysis of individual patient data, recently called for the NASCET method to be adopted as the standard method because of the potential for confusion.3 Our data show that uniformity of measurement methods is indeed desirable because the use of different methods also potentially results in significant differences in the prevalence of surgically relevant severe stenosis, even allowing for different thresholds for the definition of severe disease. In regard to which method should be adopted, we have shown that overall, no method was clearly superior to any other and all 3 methods are acceptably valid with DSA. However, because the prognostic value of the CC method has been shown indirectly through regression analysis rather than based on primary trial data, we agree with Eliasziw et al and question the value of using this method for decision-making.18 The CC method in our study did not offer any advantages in terms of reproducibility for DSA and CEMRA. We believe that given the strong trend toward noninvasive techniques, the lower sensitivity for detection of severe stenosis with the ECST method with CEMRA provides small but significant support for the NASCET method to be adopted as gold standard, especially for centers that use CEMRA. This is the most logical alternative given that the most compelling body of evidence for guiding clinical practice for carotid disease, the pooled analysis of individual patient data from the major carotid endarterectomy trials, has been analyzed according to the NASCET method.3 However, individual centers that routinely use CC or ECST need to be familiar with the NASCET method and associated potential sources of error, and to validate these measurement methods locally in terms of accuracy and interobserver variability before introducing any changes. Similarly, our study highlights that other noninvasive modalities such as computed tomography angiography need to be formally validated according to the measurement method used and that the assumption that all measurement methods are equal if different thresholds are adopted may not necessarily hold true across different imaging modalities.
- Received January 16, 2004.
- Revision received April 21, 2004.
- Accepted June 1, 2004.
Barnett HJ, Taylor DW, Eliasziw M, Fox AJ, Ferguson GG, Haynes RB, Rankin RN, Clagett GP, Hachinski VC, Sacket DL, Thorpe KE, Meldrum HE, Spence JD. Benefit of carotid endarterectomy in patients with symptomatic moderate or severe stenosis. North American Symptomatic Carotid Endarterectomy Trial Collaborators. N Engl J Med. 1998; 339: 1415–1425.
Rothwell PM, Eliasziw M, Gutnikov SA, Fox AJ, Taylor DW, Mayberg MR, Warlow CP, Barnett HJ, Carotid Endarterectomy Trialist Collaboration. Analysis of pooled data from the randomised controlled trials of endarterectomy for symptomatic carotid stenosis. Lancet. 2003; 361: 107–116.
Rothwell PM, Gibson RJ, Slattery J, Warlow CP. Prognostic value and reproducibility of measurements of carotid stenosis. A comparison of three methods on 1001 angiograms. European Carotid Surgery Trialists’ Collaborative Group. Stroke. 1994; 25: 2440–2444.
Wardlaw JM, Lewis SC, Humphrey P, Young G, Collie D, Warlow CP. How does the degree of carotid stenosis affect the accuracy and interobserver variability of magnetic resonance angiography? J Neurol Neurosurg Psychiatry. 2001; 71: 155–160.
Rothwell PM, Gibson RJ, Slattery J, Sellar RJ, Warlow CP. Equivalence of measurements of carotid stenosis. A comparison of three methods on 1001 angiograms. European Carotid Surgery Trialists’ Collaborative Group. Stroke. 1994; 25: 2435–2439.
Berry E, Kelly S, Westwood ME, Davies LM, Gough MJ, Bamford JM, Meaney JF, Airey CM, Cullingworth J, Barbieri M, Jackson A, Smith MA. The cost-effectiveness of magnetic resonance angiography for carotid artery stenosis and peripheral vascular disease: a systematic review. Health Technol Assess. 2002; 6: 1–155.
U-King-Im JM, Trivedi RA, Graves MJ, Higgins NH, Cross JJ, Tom BD, Hollingworth W, Eales H, Warburton EA, Kirkpatrick PJ, Antoun NM, Gillard JH. Contrast-enhanced MR angiography for carotid disease: diagnostic accuracy and potential therapeutic impact. Neurology. 2004; 62: 1282–1290.
Cosottini M, Pingitore A, Puglioli M, Michelassi MC, Lupi G, Abbruzzese A, Calabrese R, Lombardi M, Parenti G, Bartolozzi C. Contrast-enhanced three-dimensional magnetic resonance angiography of atherosclerotic internal carotid stenosis as the noninvasive imaging modality in revascularization decision making. Stroke. 2003; 34: 660–664.
Remonda L, Senn P, Barth A, Arnold M, Lovblad KO, Schroth G. Contrast-enhanced 3D MR angiography of the carotid artery: comparison with conventional digital subtraction angiography. AJNR Am J Neuroradiol. 2002; 23: 213–219.
Nederkoorn PJ, van der Graaf Y, Eikelboom BC, van der Lugt A, Bartels LW, Mali WP. Time-of-flight MR angiography of carotid artery stenosis: does a flow void represent severe stenosis? AJNR Am J Neuroradiol. 2002; 23: 1779–1784.
Vanninen R, Manninen H, Koivisto K, Tulla H, Partanen K, Puranen M. Carotid stenosis by digital subtraction angiography: reproducibility of the European Carotid Surgery Trial and the North American Symptomatic Carotid Endarterectomy Trial measurement methods and visual interpretation. AJNR Am J Neuroradiol. 1994; 15: 1635–1641.
Young GR, Humphrey PR, Nixon TE, Smith ET. Variability in measurement of extracranial internal carotid artery stenosis as displayed by both digital subtraction and magnetic resonance angiography: an assessment of three caliper techniques and visual impression of stenosis. Stroke. 1996; 27: 467–473.
Young GR, Humphrey PR, Shaw MD, Nixon TE, Smith ET. Comparison of magnetic resonance angiography, duplex ultrasound, and digital subtraction angiography in assessment of extracranial internal carotid artery stenosis. J Neurol Neurosurg Psychiatry. 1994; 57: 1466–1478.
Young GR, Sandercock PA, Slattery J, Humphrey PR, Smith ET, Brock L. Observer variation in the interpretation of intra-arterial angiograms and the risk of inappropriate decisions about carotid endarterectomy. J Neurol Neurosurg Psychiatry. 1996; 60: 152–157.
Eliasziw M, Smith RF, Singh N, Holdsworth DW, Fox AJ, Barnett HJ. Further comments on the measurement of carotid stenosis from angiograms. North American Symptomatic Carotid Endarterectomy Trial (NASCET) Group. Stroke. 1994; 25: 2445–2449.