Preoperative Diagnosis of Carotid Artery Stenosis
Accuracy of Noninvasive Testing
Background and Purpose— Carotid endarterectomy has been shown to be beneficial in symptomatic patients with a severe stenosis (70% to 99%) of the internal carotid artery (ICA). Digital subtraction angiography (DSA) is the standard of reference in the diagnosis of carotid artery stenosis but has a relatively high complication rate. In a diagnostic study we investigated the accuracy of noninvasive testing compared with DSA.
Methods— In a prospective diagnostic study we performed duplex ultrasound (DUS), magnetic resonance angiography (MRA), and DSA on 350 consecutive symptomatic patients. Stenoses were measured with the observers blinded for clinical information and other test results. Separate and combined test results of DUS and MRA were compared with the reference standard DSA. Only the stenosis measurements of the arteries on the symptomatic side were included in the analyses.
Results— DUS analyzed with previously defined criteria resulted in a sensitivity of 87.5% (95% CI, 82.1% to 92.9%) and a specificity of 75.7% (95% CI, 69.3% to 82.2%) in identifying severe ICA stenosis (70% to 99%). Stenosis measurements on MRA yielded a sensitivity of 92.2% (95% CI, 86.2% to 96.2%) and a specificity of 75.7% (95% CI, 68.6% to 82.5%). When we combined MRA and DUS results, agreement between these 2 modalities (84% of patients) gave a sensitivity of 96.3% (95% CI, 90.8% to 99.0%) and a specificity of 80.2% (95% CI, 73.1% to 87.3%) for identifying severe stenosis.
Conclusions— MRA showed a slightly better accuracy than DUS in the diagnosis of carotid artery stenosis. To achieve the best accuracy, however, both tests should be performed subsequently.
- angiography, digital subtraction
- carotid endarterectomy
- carotid stenosis
- magnetic resonance angiography
- ultrasonography, Doppler, duplex
Carotid endarterectomy was shown to be beneficial in symptomatic patients with a severe stenosis (70% to 99%) of the internal carotid artery (ICA) in 2 large randomized trials.1–4⇓⇓⇓ Subgroups of patients with a 50% to 69% stenosis may also expect a small benefit from carotid endarterectomy. The diagnosis of severe stenosis (70% to 99%), however, remains crucial for the majority of patients. Increasing degree of stenosis yielded increasing benefit from surgery, making precise estimation of the degree of stenosis very important. In the trials the degree of stenosis was assessed with digital subtraction angiography (DSA), which consequently has become the standard of reference for selecting patients for carotid surgery. DSA, however, has a risk of morbidity and mortality, which decreases the potential overall benefit of endarterectomy. In the literature a risk of 4% of transient ischemic attack (TIA) or minor stroke and 1% of major stroke and even a small risk of death (<1%) have been reported.5,6⇓ More recently, however, a lower rate of neurological complications due to DSA was reported: 0.5% for stroke and 0.4% for TIA.7 On the other hand, even patients without apparent neurological complications after DSA have been shown to develop minor asymptomatic infarctions due to microembolisms.8
Over the last decade many diagnostic studies have been published in which noninvasive diagnostic tests such as duplex ultrasound (DUS) and magnetic resonance angiography (MRA) or combinations of these tests were compared with DSA.9–19⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓ Two meta-analytic reviews have been published summarizing the literature on the diagnostic performance of DUS and MRA from before 1996. One concluded that the actual sensitivity and specificity for MRA remain unknown but that these are probably lower than reported in the literature because of the presence of verification bias and because frequently both carotid arteries (symptomatic and asymptomatic) were included in the analyses.20 The other meta-analysis reported that noninvasive testing at that point did not appear to be an adequate substitute for DSA for patients about to have carotid endarterectomy.21 Thereafter, the noninvasive imaging techniques have continued to develop. A review of previous publications on this topic published during 1993–1998, however, criticized the design of the studies and proposed guidelines for diagnostic studies on carotid artery imaging.22 Accordingly, a recent review summarizing publications during 1990–1999 concluded that MRA seemed accurate for selecting patients for carotid endarterectomy but that evidence was not very robust because of the heterogeneity of the studies included.23 The need for a prospective diagnostic study on noninvasive testing was recently recognized in the literature.13
The objective of this study was to obtain reliable estimates of the diagnostic accuracy of DUS, MRA, and a combination of these tests compared with DSA as reference standard.
Subjects and Methods
From January 1997 to November 2000, 350 consecutive symptomatic patients suspected of having carotid artery stenosis were included in a prospective diagnostic study. Patients underwent DUS, MRA, and DSA examination within a maximum of 4 weeks. All patients had experienced symptoms of carotid artery disease (TIA, minor disabling ischemic stroke, or amaurosis fugax) in the prior 6 months. Patients underwent complete neurological examination within 24 hours before and after DSA to establish possible deficits caused by this procedure by an independent physician (P.J.N. or O.E.H.E.). In case of a possible complication, a senior neurologist was consulted (L.J.K.). We excluded patients with contraindications for MRA such as claustrophobia or metal implants not suitable for MR examination. Medical history was recorded from all patients. The decision of whether or not to perform carotid endarterectomy was made in the clinical setting on the basis of the DSA examinations. The frequency of carotid endarterectomy in our study population and the complication rate within 4 weeks after surgery were recorded by the vascular surgeons. In case of a possible complication, a senior neurologist was consulted. Patients were enrolled in the University Medical Center Utrecht, University Medical Center Rotterdam, and Enschede Medical Center (Netherlands). Our study was approved by a medical ethical committee, and all patients gave their written informed consent. We met recently published quality criteria for design and presentation of diagnostic studies on carotid artery imaging.22
The degree of stenosis on DUS was determined on the basis of the peak systolic velocity (PSV) in the proximal part of the ICA. The PSV is considered the most accurate estimator of the degree of stenosis for DUS.24 We validated DUS results in a pilot series before the present study started. By means of receiver operating characteristic (ROC) curves, we previously defined optimal cutoff criteria for the PSV for different stenosis categories (Table 1). In these criteria the threshold of 70% stenosis is represented by a PSV of 270 cm/s.25
DSA was performed by selective positioning of an intra-arterial catheter in both common carotid arteries. From each carotid bifurcation, 3 projections (lateral, posteroanterior, and oblique) were acquired. Additional projections of occasionally performed rotational DSA examinations were not used in the context of this study.
In all 3 hospitals, MRA was performed on a 1.5-T MRI system, with the use of a 3-dimensional time-of-flight technique. Postprocessing subvolumes were generated to visualize each carotid bifurcation and to create maximum intensity projection (MIP) images. The DSA and MRA protocols have been described in detail elsewhere.26
The DSA and MRA test results were read by 1 observer for each hospital (A.F.W. for Utrecht and Enschede, A. van der L. for Rotterdam). The observers were blinded for clinical information and for the results of the other tests. The DSA and MRA images were read independently with a period of at least 1 month between the readings. The observers read the DSA and MRA on printed hard copies. For MRA we only used MIP images. The grade of stenosis on both DSA and MRA was measured according to the North American Symptomatic Carotid Endarterectomy Trial (NASCET) criteria.1 The degree of stenosis is defined as the remaining lumen at the stenosis as percentage of the normal lumen distal to the stenosis. For a valid comparison with DSA, we used only the percentage of stenosis measured on lateral, posteroanterior, and oblique projections on MRA. The maximum of these 3 measurements, on both DSA and MRA, was used in the analyses.26 To estimate the reproducibility, the percentage of stenosis was measured by 2 independent observers (A.F.W. and P.C.B.) for a representative sample of 170 patients on both DSA and MRA. Figures 1 and 2⇓ show examples of moderate and severe stenoses, respectively.
Test results of DUS and MRA were first analyzed separately compared with the reference standard DSA.27 The measured stenoses were divided in categories (0% to 29%, 30% to 49%, 50% to 69%, 70% to 99%, 100%). We included for each patient only the estimate of the stenosis of the carotid artery on the symptomatic side in the analyses. Results were interpreted by calculating sensitivity, specificity, and positive and negative predictive values, defining severe stenosis (70% to 99%) on DSA as a positive test result. Second, DUS and MRA results were combined and considered as a combination test. We analyzed the part in which DUS and MRA were in agreement concerning the diagnosis of severe stenosis (70% to 99%) as a separate group. The combined results of this group were again compared with DSA. We calculated kappa (κ) statistics for the DSA and MRA results of the 170 patients read by 2 observers.
Of the 350 patients included, 249 patients were enrolled in Utrecht, 62 in Rotterdam, and 39 in Enschede. The baseline characteristics and relevant medical history are listed in Table 2. To assess generalizability, we monitored the reasons for exclusion in 1 hospital (Utrecht). In this hospital, during the study period, 297 patients underwent DSA to decide on carotid endarterectomy. Of this total, 84% (249 patients) were included in the study. Reasons for exclusion were claustrophobia in 3.4%, metal implant not suitable for MR examination in 3.0%, and refusal to participate in the study in 8.8%. Reasons for refusal were mostly stress for planned surgery or participation in other studies. Baseline characteristics of the 48 excluded patients did not differ significantly from the included population.
From the total of 350 patients included in the 3 hospitals, the following numbers of stenosis measurements from the symptomatic side were interpretable and could be included in the analyses: DSA 323, DUS 330, and MRA 295. Missing values were caused by the following reasons: sometimes it was not feasible to perform all 3 tests before surgery, some patients withdrew from the study after 1 or 2 tests, and the test was not always correctly performed according to our study protocol (see Subjects and Methods). In DUS occasionally the PSV was not measured. Finally, it was impossible to measure stenosis because of poor quality and reliability of the MRA recordings in 10 patients and of the DSA recordings in 7 patients. The complication rate of DSA in our series was 1.4% minor stroke (95% CI, 0.1% to 3.3%), 0.3% major stroke (95% CI, 0.0% to 1.6%), and 0.6% mortality (95% CI, 0.1% to 2.0%). Two hundred twenty patients underwent carotid endarterectomy (63%). The complication rate of surgery (within 4 weeks) was 3.2% minor stroke (95% CI, 1.3% to 6.5%) and 0.5% major stroke (95% CI, 0.0% to 2.5%).
Diagnostic Test Results
In Table 3, the test results of DUS are presented with DSA as reference. DUS analyzed with previously defined PSV criteria resulted in a sensitivity of 87.5% (95% CI, 82.1% to 92.9%) and a specificity of 75.7% (95% CI, 69.3% to 82.2%) in identifying severe ICA stenosis (70% to 99%). Stenosis measurements (NASCET) on MRA compared with DSA yielded a sensitivity of 92.2% (95% CI, 86.2% to 96.2%) and a specificity of 75.7% (95% CI, 68.6% to 82.5%) (Table 4). The positive predictive value was 75.4% (95% CI, 68.9% to 82.0%) for DUS and 76.3% (95% CI, 69.6% to 83.0%) for MRA, and the negative predictive value was 87.7% (95% CI, 82.3% to 93.0%) for DUS and 92.0% (95% CI, 85.8% to 96.1%) for MRA. Both DUS and MRA had a tendency to overestimate the degree of stenosis compared with DSA. In 274 patients, stenosis measurements from all 3 test results (DSA, DUS, and MRA) were available. In 229 of these patients (84%), DUS and MRA were in agreement in regard to diagnosing severe stenosis (70% to 99%). In this subgroup, with agreement between the 2 modalities, the combination of MRA and DUS results (ie, considered as a single test) gave a sensitivity of 96.3% (95% CI, 90.8% to 99.0%) and a specificity of 80.2% (95% CI, 73.1% to 87.3%) for identifying severe stenosis (Table 5). The positive predictive value for the combination was 81.2% (95% CI, 74.5% to 88.0%), and the negative predictive value was 96.0% (95% CI, 90.2% to 98.9%). With the use of this approach, the diagnosis concerning severe stenosis remained unclear in case of disagreement between DUS and MRA (16% of the patients). In this combination strategy there was also a tendency to overestimate the stenosis compared with the reference test DSA. In all 24 cases of overestimation in this combination strategy, on the basis of concordant results, DSA categorized the stenosis only 1 category lower as 50% to 69%. The diagnostic test results of the single tests and combination are listed in Table 6. Interobserver variability for DSA and MRA was very good and was similar for both tests (κ=0.79 [95% CI, 0.74 to 0.84] for DSA and κ=0.79 [95% CI, 0.73 to 0.84] for MRA).
MRA showed a slightly better accuracy than DUS in the diagnosis of carotid artery stenosis. In the combination test results, agreement in diagnosing severe stenosis (70% to 99%) between DUS and MRA (84% of the patients) yielded a high sensitivity and specificity. However, with the use of this approach, the diagnosis remains unclear in case of clear disagreement. Both DUS and MRA have the tendency to overestimate stenosis compared with the standard of reference DSA.
DUS is commonly used to screen patients with possible carotid artery disease11 but has also been suggested as the sole test to select patients for carotid endarterectomy.12,16⇓ This technique, however, has limitations, such as variability in optimal thresholds,25 possible effect of verification bias,28 and limited morphological information. MRA can also be used as a noninvasive test to diagnose carotid artery stenosis.20,21⇓ MRA provides more morphological information without exposure to the risks of DSA. At the start of our study the time-of-flight technique was the state-of-the-art technique for MRA. MRA techniques, however, have improved during the period of our study. The introduction of contrast-enhanced MRA may add to the development.29,30⇓ Contrast-enhanced MRA provides additional morphological information about the origin of the carotid arteries and intracranial vessels, and the effect of flow-related artifacts occurring with the time-of-flight technique is diminished. To date, however, only a few studies have been published reporting the results of contrast-enhanced MRA validated against DSA in small cohorts, precluding a precise estimate of its accuracy.23 For a valid evaluation of a new imaging technique, an adequately powered study is mandatory.22 However, such a study is likely to be large, expensive, and time consuming, precluding application of the newest imaging protocols. Recently, however, we also introduced contrast-enhanced MRA in our clinical setting supplemental to the time-of-flight protocol, allowing us to estimate its accuracy in a subgroup of patients.
Irrespective of the use of intravenous contrast, very good accuracies have been published for MRA if used in combination with DUS.13,14,17,19⇓⇓⇓ Most of the studies on this subject, however, did not meet all standard criteria for design and reporting of the diagnostic tests.22 The number of patients that underwent MRA in these series was relatively small, and often the data were recorded retrospectively, introducing the risk of observer bias.
Our reported accuracies of DUS and MRA might seem relatively low compared with other studies.11–21⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓ However, we believe that our prospective design added to valid and unbiased estimates. Another explanation is the fact that we only included the test results of the carotid artery on the symptomatic side in the analyses, yielding lower accuracies. Furthermore, because the decision of whether to perform endarterectomy needs to be made in regard to the symptomatic artery, excluding the asymptomatic side reflects clinical practice. Previously published reports generally also included the asymptomatic side, which was not suspected of having stenosis on DUS. The majority of the arteries on the asymptomatic side show a stenosis percentage far below the 70% threshold or no stenosis at all, making it more likely for the different tests to agree. In this way the number of true negative results is inflated, and thus specificity may be overestimated in those studies. In our data the specificities increased by 10.7% for DUS and 14.2% for MRA if the stenosis measurements of all arteries were included.
Furthermore, it is important to realize that diagnostic testing is limited by some general methodological limitations. In a diagnostic study a new test by definition never exactly agrees with the reference test.27 Even if readings of the reference test itself were to be repeated, there would always be a certain variability in results. The aim of a diagnostic study should not be to achieve the highest possible accuracy. The more relevant question is to what degree a new test under investigation differs from the reference test and what implications this has for the outcome of clinical decisions for individual patients. For this purpose the most realistic estimate of the accuracy is requested. The results should be the guideline for deciding on diagnostic strategies in the clinical setting. Furthermore, to make the right policy decisions from a societal perspective, in addition to a valid estimate of the accuracy of noninvasive testing, cost-effectiveness should be taken into account.
Both DUS and MRA had a tendency to overestimate the degree of stenosis. In DUS verification bias may have played a role. Verification bias may exist if the decision to perform the gold standard procedure depends on the results of the test under investigation.28 The sensitivity may be lower, and specificity may be higher, after adjustment for this bias. In our study patients were often screened with DUS in the clinical setting before inclusion. On the basis of ethical grounds, inclusion in the study depended on the decision of the clinician to perform DSA if carotid endarterectomy was considered. Accuracy of DUS related to the 70% stenosis threshold was estimated afterward among patients selected for DSA. The tabulations show that all categories of degree of stenosis are present, although the majority have a moderate or severe stenosis. In our opinion, however, it is precisely this selected group of patients, suspected of having ICA stenosis at DUS examination, that constitutes the right domain to answer our study objective, reflecting the population for which the decision on surgery must be made in daily clinical practice.
MRA may also overestimate stenosis in comparison with DSA. Overestimation on MRA may occur when all 12 available projections are used for the stenosis measurements and are subsequently compared with DSA, on which stenosis is often measured in only 3 directions (lateral, posteroanterior, and oblique).26 In our study, with regard to the endarterectomy trials, in all patients we assessed the degree of stenosis on DSA using the 3 standard directions, and therefore we used only the same 3 corresponding projections on MRA. Nevertheless, overestimation still occurred. Although we have interpreted our findings as overestimation on MRA, it is very possible that DSA underestimates the true degree of stenosis and that new (3-dimensional) techniques can estimate the degree of stenosis more precisely.
Using the combination strategy of DUS and MRA yielded the highest accuracy. An important finding was that in all cases in which both DUS and MRA overestimated the stenosis (and agreed that carotid endarterectomy was indicated), the stenosis was classified in the 50% to 69% category according to DSA. On the basis of recently published results from NASCET and the European Carotid Surgery Trial, these patients still have limited benefit from carotid endarterectomy.3,4⇓ It is expected that in the near future additional evidence will become available regarding which patients may expect the most benefit from carotid endarterectomy in the 50% to 69% stenosis category.31,32⇓
In regard to the reported complication rates of DSA and of carotid endarterectomy, in general, the number of complications is relatively small compared with the total population. Moreover, the confidence intervals of the rates in our population of 350 patients are broad. Therefore, a significant difference with complication rates reported in literature cannot be proven. We reported 2 deaths after DSA. One patient experienced a major stroke during the procedure and died 72 hours later. The other patient suffered from generalized embolic disease and cardiac events before and after the DSA procedure and died 5 days after DSA after his symptoms had worsened. It remains unsure whether his death had a direct causal relation to the DSA procedure. The complication rates of carotid endarterectomy were recorded over period of 4 weeks after surgery. We realize that minor events that occurred after the patients were dismissed from the hospital might be underreported.
In conclusion, MRA showed a slightly better accuracy than DUS in the diagnosis of carotid artery stenosis. DUS test results might be influenced by verification bias. Furthermore, both tests have a tendency to overestimate the degree of stenosis compared with the reference test DSA. The best accuracy is achieved in case of agreement between the 2 tests. Therefore, in our opinion, in a noninvasive diagnostic strategy both DUS and MRA should be performed subsequently. To make the right policy decisions, however, in addition to a valid estimate of the accuracy of noninvasive testing, cost-effectiveness should be taken into account.
This study was funded by a grant (OG/030) from the Dutch Ministry of Health, Welfare, and Sports. Participants in the diagnostic study were as follows: University Medical Center Utrecht: P. Nederkoorn and Y. van der Graaf (Julius Center for Patient Oriented Research), O. Elgersma and W. Mali (Department of Radiology), B. Eikelboom and J. Blankensteijn (Department of Vascular Surgery), L.J. Kappelle (Department of Neurology); Erasmus Medical Center Rotterdam: M. Hunink (Department of Epidemiology and Biostatistics and Department of Radiology), H. van Urk (Department of Vascular Surgery), D. Dippel (Department of Neurology), H. Pieterman (Department of Radiology); Medical Center Enschede: R. van Det, R. Brouwer (Department of Vascular Surgery), A. Huisman (Department of Radiology), C. Leijzer (Department of Neurology).
- Received February 14, 2002.
- Revision received April 5, 2002.
- Accepted April 22, 2002.
- ↵Barnett HJ, Taylor DW, Eliasziw M, Fox AJ, Ferguson GG, Haynes RB, Rankin RN, Clagett GP, Hachinski VC, Sackett DL, Thorpe KE, Meldrum HE, for the North American Symptomatic Carotid Endarterectomy Trial Collaborators. Benefit of carotid endarterectomy in patients with symptomatic moderate or severe stenosis. N Engl J Med. 1998; 339: 1415–1425.
- ↵Hankey GJ, Warlow CP, Molyneux AJ. Complications of cerebral angiography for patients with mild carotid territory ischaemia being considered for carotid endarterectomy. J Neurol Neurosurg Psychiatry. 1990; 53: 542–548.
- ↵Davies KN, Humphrey PR. Complications of cerebral angiography in patients with symptomatic carotid territory ischaemia screened by carotid ultrasound. J Neurol Neurosurg Psychiatry. 1993; 56: 967–972.
- ↵Johnston DC, Chapman KM, Goldstein LB. Low rate of complications of cerebral angiography in routine clinical practice. Neurology. 2001; 57: 2012–2014.
- ↵Wardlaw JM, Lewis SC, Humphrey P, Young G, Collie D, Warlow CP. How does the degree of carotid stenosis affect the accuracy and interobserver variability of magnetic resonance angiography? J Neurol Neurosurg Psychiatry. 2001; 71: 155–160.
- ↵Eliasziw M, Rankin RN, Fox AJ, Haynes RB, Barnett HJ, for the North American Symptomatic Carotid Endarterectomy Trial (NASCET) Group. Accuracy and prognostic consequences of ultrasonography in identifying severe carotid artery stenosis. Stroke. 1995; 26: 1747–1752.
- ↵Jackson MR, Chang AS, Robles HA, Gillespie DL, Olsen SB, Kaiser WJ, Goff JM, O’Donnell SD, Rich NM. Determination of 60% or greater carotid stenosis: a prospective comparison of magnetic resonance angiography and duplex ultrasound with conventional angiography. Ann Vasc Surg. 1998; 12: 236–243.
- ↵Johnston DC, Goldstein LB. Clinical carotid endarterectomy decision making: noninvasive vascular imaging versus angiography. Neurology. 2001; 56: 1009–1015.
- ↵Kent KC, Kuntz KM, Patel MR, Kim D, Klufas RA, Whittemore AD, Polak JF, Skillman JJ, Edelman RR. Perioperative imaging strategies for carotid endarterectomy: an analysis of morbidity and cost-effectiveness in symptomatic patients. JAMA. 1995; 20: 274: 888–893.
- ↵Kagawa R, Moritake K, Shima T, Okada Y. Validity of B-mode ultrasonographic findings in patients undergoing carotid endarterectomy in comparison with angiographic and clinicopathologic features. Stroke. 1996; 27: 700–705.
- ↵Kallmes DF, Omary RA, Dix JE, Evans AJ, Hillman BJ. Specificity of MR angiography as a confirmatory test of carotid artery stenosis. AJNR Am J Neuroradiol. 1996; 17: 1501–1506.
- ↵Rothwell PM, Pendlebury ST, Wardlaw J, Warlow CP. Critical appraisal of the design and reporting of studies of imaging and measurement of carotid stenosis. Stroke. 2000; 31: 1444–1450.
- ↵Westwood ME, Kelly S, Berry E, Bamford JM, Gough MJ, Airey CM, Meaney JF, Davies LM, Cullingworth J, Smith MA. Use of magnetic resonance angiography to select candidates with recently symptomatic carotid stenosis for surgery: systematic review. BMJ. 2002; 324: 198.
- ↵Elgersma OE, van Leersum M, Buijs PC, van Leeuwen MS, van de Schouw YT, Eikelboom BC, van der G. Changes over time in optimal duplex threshold for the identification of patients eligible for carotid endarterectomy. Stroke. 1998; 29: 2352–2356.
- ↵Elgersma OE, Wust AF, Buijs PC, van der GY, Eikelboom BC, Mali WP. Multidirectional depiction of internal carotid arterial stenosis: three-dimensional time-of-flight MR angiography versus rotational and conventional digital subtraction angiography. Radiology. 2000; 216: 511–516.
- ↵Phan T, Huston J III, Bernstein MA, Riederer SJ, Brown RD Jr. Contrast-enhanced magnetic resonance angiography of the cervical vessels: experience with 422 patients. Stroke. 2001; 32: 2282–2286.