Duplex Ultrasound Criteria for the Identification of Carotid Stenosis Should Be Laboratory Specific
Background and Purpose Published criteria for the determination of carotid stenosis have been widely applied by vascular laboratories. We compared two vascular laboratories and their duplex ultrasound (DU) machines in terms of their overall diagnostic performance and the optimal criteria to identify patients who have a 70% to 99% stenosis of the internal carotid artery.
Methods Measurements of stenosis by DU and angiography were compared for 123 carotid arteries (60 arteries, laboratory A; 63 arteries, laboratory B). Receiver operating characteristic (ROC) curves were created, and the areas under the ROC curves and the optimal criteria for determining a 70% to 99% stenosis were compared. Multiple regression analysis was used to measure the effect of laboratory on the relationship between angiographic stenosis and DU velocity parameters.
Results Areas under the ROC curves were similar for both laboratories (0.89 to 0.90, laboratory A; 0.90 to 0.92, laboratory B). However, the optimal criterion for the identification of a 70% to 99% carotid stenosis was different for each laboratory. For most velocity parameters, based on regression analyses, the predicted percent angiographic stenosis for laboratory A was significantly greater than that for laboratory B. In addition, performance differed between the laboratories when established criteria from the literature were applied.
Conclusions Two vascular laboratories with similar diagnostic accuracy by ROC analysis have markedly different “optimal” DU criteria. For a given angiographic stenosis, velocities in one laboratory were consistently greater than those in the other laboratory. Laboratory-specific criteria rather than published criteria should be used to identify patients with internal carotid artery stenoses.
Duplex ultrasonography is the established method for screening patients with suspected carotid stenosis. Because of the cost, inconvenience, and risk associated with contrast angiography, a number of centers advocate the primary use of DU for the preoperative evaluation of patients for carotid endarterectomy.1 2 3 4 5 6 Recently, we found DU to be a potentially cost-effective alternative to contrast angiography in the preoperative evaluation of patients with symptomatic carotid stenosis.6 Because of the frequent use of DU in the diagnosis and management of carotid disease, variations in diagnostic performance among laboratories or DU devices can have a significant impact on the outcomes of patients with carotid stenosis.
Typically, velocities measured within the CCA and ICA are used to establish criteria for determining a particular degree of carotid stenosis. These “criteria” are chosen to allow optimization of a desired end point, such as diagnostic accuracy or stroke reduction. For example, Moneta et al,7 on the basis of achieving maximal accuracy, found that a 70% to 99% angiographic stenosis was best predicted by a ratio of the ICA PSV to the CCA PSV greater than 4.0.
The results from NASCET8 prompted a number of investigators to identify appropriate DU criteria for detecting a 70% to 99% carotid stenosis.7 9 10 11 However, these reports fail to reach consensus; for example, for ICA PSV, Moneta et al7 found that readings >325 cm/s resulted in the greatest accuracy, whereas Faught et al9 found that an ICA PSV >210 cm/s maximized accuracy. Application of the ICA PSV criteria found by Faught et al to the laboratory data presented by Moneta et al results in a decrease in accuracy from 88.0% to 83.7%. We sought to further understand the reasons for these differences by comparing five DU velocity parameters with angiographic results for two neighboring but distinctly separate laboratories in terms of their abilities to optimally predict a 70% to 99% angiographic stenosis.
Subjects and Methods
Over a 19-month period, we studied 32 patients at the Beth Israel Hospital and 32 patients at the Brigham and Women’s Hospital who were referred for evaluation of carotid stenosis by contrast angiography. All patients were initially studied with DU and then by contrast angiography. The average patient age was 70 years (range, 48 to 87 years). There were 42 men and 22 women. Risk factors for atherosclerotic occlusive disease included hypertension (73%), smoking (64%), diabetes (35%), and hypercholesterolemia (47%). Of the 128 arteries evaluated, 123 were imaged by both techniques (4 arteries were inadequately imaged by contrast angiography and 1 was inadequately imaged by DU). Fifty-four arteries were symptomatic and 69 were asymptomatic.
DU was performed by certified vascular technologists in both laboratories (one in laboratory A, two in laboratory B). Examinations were performed using either an Acuson 128 (laboratory A) or an ATL Ultramark 9 (Advanced Technology Laboratories; laboratory B) color duplex scanner. Velocity waveforms were sequentially obtained from the CCA and ICA. The EDV and PSV were recorded at each of the above locations. In both laboratories, the angle between the ultrasound beam and the direction of blood flow was maintained at ≤60°. Sampling techniques were similar in terms of the frequency of measurements and the position of the Doppler cursor. High carrier frequency linear-array transducers were used in both laboratories. The frequency of the Doppler probe varied between laboratories (3.5 MHz, laboratory A; 5 MHz, laboratory B), as did the method used to align the flow-velocity vector (laboratory A used the color flow image and laboratory B used the B-mode image). A markedly diminished velocity implied high-grade stenosis, and the absence of flow implied occlusion.
Digital subtraction arteriograms included at least two views of the carotid bifurcation and intracranial vessels. Images were traced from an overhead projector and then measured with digital calipers (Fowler). Percent stenosis was determined in the manner previously described by NASCET.8 We used angiographic measurements from a single reviewer for our analysis and checked the reliability of this assignment with readings from a second reviewer (κ=0.93).
ROC analysis12 was used to calculate sensitivities and specificities of DU for detecting a 70% to 99% stenosis of the carotid artery using contrast angiography as the gold standard. For this analysis, we assumed that all arteries were independent.13 We evaluated five different duplex velocity parameters: (1) ICA PSV, (2) ICA EDV, (3) ICA/CCA PSV, (4) ICA/CCA EDV, and (5) (ICA PSV)/(CCA EDV). The areas under the ROC curves were obtained by the trapezoidal method,14 which is a nonparametric procedure that does not require any assumptions regarding the functional form of the ROC curve.
Multiple regression analyses were performed to determine the effect of the vascular laboratory on the association between percent angiographic stenosis and each of the DU velocity parameters. We used a forward stepwise procedure to fit regression equations, with percent angiographic stenosis as the dependent variable and each of the five aforementioned velocity parameters as the independent variable. To account for potential curvilinear relationships between the velocity parameters and percent angiographic stenosis, we allowed for squared and cubed velocity terms to enter the regression model. The independent variable of interest was an indicator variable for DU laboratory. A value of P<.05 was considered significant.
Optimal duplex criteria for designating a DU test result as positive were determined separately for the two laboratories using the five duplex velocity parameters as continuous variables (ie, velocities were not categorized). We used two end points for selecting the optimal criteria for the identification of a 70% to 99% angiographic stenosis. First, we chose a criterion that resulted in the greatest accuracy. For every possible velocity cut point (ie, an ICA PSV cut point of 200 cm/s indicates a criterion of >200 cm/s for calling a test positive), there is an associated sensitivity, specificity, and accuracy. To determine the optimal duplex criterion, we chose the velocity cut point that was associated with the greatest accuracy. As a second approach, we selected the criterion that minimized the probability of stroke at 2 years for symptomatic patients.15 We have created and previously reported a model that allows analysis of stroke as an end point.6 This analysis is based on the fact that a noninvasive test, when used preoperatively for carotid endarterectomy, might result in late strokes as a result of inappropriate treatment dictated by inaccurate test results. For the stroke minimization analysis, we assigned 2-year stroke risks of 12.6%, 27.6%, 7.6%, and 10.8% for true-positive, false-negative, true-negative, and false-positive results, respectively.6 8 Bootstrap confidence intervals16 were calculated for the differences between the cut points associated with each of the optimal criteria in the two laboratories. One-sided confidence intervals were calculated on the basis of the results from the multiple regression analyses (ie, a positive regression parameter estimate for laboratory A corresponds to a lower cut point for the laboratory A DU device; see Fig 1⇓). In addition, we applied published criteria for the determination of a 70% to 99% carotid stenosis to each DU device and compared the resulting sensitivities and specificities.
Of the 60 carotid arteries imaged by DU in laboratory A, 33 (55.0%), 20 (33.3%), and 7 (11.7%) had 0% to 69%, 70% to 99%, and 100% angiographic stenoses, respectively. Of the 63 carotid arteries imaged by DU in laboratory B, 36 (57.1%), 19 (30.2%), and 8 (12.7%) had 0% to 69%, 70% to 99%, and 100% angiographic stenoses, respectively. One angiographically occluded artery was misdiagnosed as patent in each laboratory.
The areas under the ROC curves for each of the five velocity parameters ranged between 0.89 and 0.90 for the laboratory A device and between 0.90 and 0.92 for the laboratory B device, depending on the velocity parameter used. For example, the areas under the ROC curves for ICA PSV were 0.90 and 0.92 for laboratories A and B, respectively. The results were similar when the patients who had occlusions were omitted (0.89 to 0.91, laboratory A; 0.90 to 0.93, laboratory B). Although the two devices were equivalent in terms of their overall diagnostic accuracy, two points along the ROC curves that corresponded to the same cut point were rarely proximate (Fig 2⇓).
Table 1⇓ shows the results from the multiple regression analyses. Significant curvilinear relationships (either quadratic or cubic) were found between percent angiographic stenosis and each of the five velocity parameters. There was a significant difference between laboratories for the following velocity parameters: ICA EDV, ICA/CCA EDV, and (ICA PSV)/(CCA EDV). A borderline significant difference was found for ICA/CCA PSV (P=.056). In addition, the direction of the effect was consistent for all of the velocity parameters. The angiographic stenosis predicted by the fitted model for laboratory A was greater than that predicted for laboratory B by an absolute increase of 5.1% for ICA PSV to 13.0% for ICA/CCA EDV.
Tables 2⇓ and 3⇓ show the optimal criteria for the identification of patients with 70% to 99% angiographic stenosis based on (1) maximization of diagnostic accuracy and (2) minimization of the expected 2-year risk of stroke for symptomatic patients. The cut points associated with the optimal DU criteria were consistently greater for laboratory B. For example, for ICA PSV, the optimal criteria were >229 cm/s for the laboratory A device and >340 cm/s for the laboratory B device for both optimization techniques. For (ICA PSV)/(CCA EDV), the optimal criteria not only differed between laboratories but also differed according to the optimization end point used. For laboratory A, the criterion that maximized diagnostic accuracy was a ratio >10.5, while the criterion that minimized the expected 2-year risk of stroke was a ratio >5.8. For laboratory B, the criterion that maximized diagnostic accuracy was a ratio >21.7, while the criterion that minimized the expected 2-year risk of stroke was a ratio >13.7. Based on one-way bootstrap confidence intervals, most of the differences in optimal criteria were significant (P<.05) or borderline significant (.05≤P<.10).
We applied three published duplex criteria for identifying 70% to 99% angiographic stenosis and found substantial variability between the sensitivity and specificity estimates for the two DU machines (Table 4⇓). For the laboratory A device, the criterion found by Hunink et al10 to best predict a 70% angiographic cutoff (ICA PSV >230 cm/s) resulted in the highest accuracy of the three criteria considered; however, applying this criterion to the laboratory B DU device resulted in the lowest accuracy of the three criteria considered. Alternatively, applying the criteria recommended by Faught et al9 (ICA PSV >130 cm/s and ICA EDV >100 cm/s) resulted in the highest accuracy for the laboratory B DU device and the lowest accuracy for the laboratory A device.
We found that the DU measurements that best predicted percent angiographic stenosis differed between two vascular laboratories. When the same velocity parameters were applied to both laboratories, the predicted angiographic stenosis for laboratory A was consistently greater than that for laboratory B for all five parameters considered. We also found that the optimal duplex criteria for identifying 70% to 99% angiographic stenosis varied with the laboratory performing the study. Specifically, for each velocity parameter, the optimal velocity cut point for the DU device in laboratory A was lower than that for the laboratory B device (Fig 1⇑). Similar observations were made by the investigators of the Asymptomatic Carotid Atherosclerosis Study,17 in which optimal ICA PSV cut points for predicting a >60% stenosis ranged from 151 to 390 cm/s among different centers.18 Our findings also explain why the recommended optimal criteria for predicting a 70% to 99% diameter narrowing of the carotid bifurcation vary so significantly among published studies.7 9 10 11
We used two end points for selecting the best criteria for each of the five velocity parameters: (1) maximization of diagnostic accuracy and (2) minimization of 2-year stroke risk for symptomatic patients. In the first case, the goal is to minimize the number of false-positive and false-negative results. This approach assumes that a false-negative result has the same importance as a false-positive result. The second approach weights each of the four possible test results (true-positive, false-negative, true-negative, and false-positive) by the expected probability of incurring a stroke by 2 years, based primarily on data published in NASCET.6 8 Although many of the studies that define optimal duplex criteria use accuracy maximization as their goal, an approach that minimizes morbidity may be more important than an approach that maximizes accuracy. Our results show that changes in the optimization approach can alter the criterion used to judge a 70% to 99% stenosis.
At least three DU criteria have been identified by various authors for determination of a 70% to 99% angiographic stenosis.7 9 10 However, application of these criteria to our data resulted in great variability in the estimates of sensitivity and specificity (Table 4⇑). For example, for laboratory A, sensitivity and specificity were 68% and 93%, respectively, for the criteria proposed by Faught et al9 (ICA PSV >130 cm/s and ICA EDV >100 cm/s) and were 100% and 83%, respectively, for the criteria proposed by Hunink et al10 (ICA PSV >230 cm/s). If the former criteria were used, there would be a substantial number of patients with surgical disease who would not be treated, whereas if the latter criterion were used, there would be a significant number of patients without disease who would be exposed to the risks of surgery. If a center were to use criteria that produced a sensitivity of 68% and a specificity of 93% for preoperative imaging of symptomatic patients, then the expected 2-year risk of stroke would be 10.9%. Alternatively, if the criteria were changed to correspond to a sensitivity of 100% and a specificity of 83%, then the expected 2-year risk of stroke for symptomatic patients would be reduced to 9.6%. In other words, use of one published criterion versus another could result in approximately 13 strokes averted, on average, for every 1000 patients evaluated with DU in laboratory A.
The source of the observed difference in diagnostic performance of the two laboratories is not clear. The patient population and the technique for performing examinations for both laboratories were similar. The angle between the ultrasound beam and the direction of blood flow was maintained at ≤60°. However, the operating frequency of the probes was different for each laboratory (3.5 MHz, laboratory A; 5 MHz, laboratory B), as was the manner in which the flow velocity vector was aligned (color flow image, laboratory A; B-mode image, laboratory B). Although these differences might affect the technical adequacy of the examination, they should not introduce bias in the estimated velocities.19 The possibility that this effect might solely be operator dependent is not supported by the interobserver variability data already published for one of the two laboratories. A net bias in velocity estimates was not seen when one observer is compared with other observers.20 Therefore, differences between DU machines seem to be the most likely explanation for the disparity between laboratories that we identified.
Differences between machines in the methodology used to generate, acquire, and angle-correct the returning echoes might be the source of this bias. Daigle et al21 showed that performance of ultrasound devices was affected by the location in the imaging field that was being sampled and the direction of the ultrasound beam. The method used to generate the DU beam relies on the sequential firing of a number of selected crystals in the transducer. This process can introduce ambiguity in the orientation of the resultant ultrasound beam. This can then cause artifactual spectral broadening of the returning echoes and tend to increase velocity estimates. Although this effect is compensated for, the method of comparison differs between manufacturers. It likely accounts for the effect we have encountered when two differing DU machines are used to evaluate carotid artery stenoses.
There are several limitations of our study. First, this was an observational study; hence, no patient was evaluated in both laboratories. However, the patient populations were similar in their clinical and demographic variables. Second, there were a limited number of patients evaluated at each laboratory, thus lowering our statistical power. However, even with a relatively small sample size, we were able to uncover an overwhelming trend across all of the five velocity parameters that we considered. For a specific angiographic stenosis, our results indicate that one laboratory consistently measured higher velocities than the other laboratory. Knowing whether this phenomenon is manufacturer specific or technologist specific would be of utmost importance for vascular laboratory directors and policy makers.
Our study supports the tenet that diagnostic criteria for determining a 70% to 99% carotid stenosis should be established for each instrument model and laboratory and not extrapolated from external studies. All vascular laboratories should be required to perform an objective evaluation of the diagnostic performance of their scanners, as well as provide definitions of optimal criteria for each relevant stenosis. In addition, more research needs to be focused on identifying those factors that are important contributors to variability in DU measurements. Adjustments then can be made when results between different scanners and/or laboratories are compared. In conclusion, we feel that it is critically important that each vascular laboratory determine its own optimal criteria and that this will have a positive impact on the care of patients with carotid disease.
Selected Abbreviations and Acronyms
|CCA||=||common carotid artery|
|ICA||=||internal carotid artery|
|NASCET||=||North American Symptomatic Carotid Endarterectomy Trial|
|PSV||=||peak systolic velocity|
|ROC||=||receiver operating characteristic|
- Received September 27, 1996.
- Revision received December 3, 1996.
- Accepted December 3, 1996.
- Copyright © 1997 by American Heart Association
Moneta GL, Edwards JM, Chitwood RW, Taylor LM, Lee RW, Cummings CA, Porter JM. Correlation of North American Symptomatic Carotid Endarterectomy Trial (NASCET) angiographic definition of 70% to 99% internal carotid artery stenosis with duplex scanning. J Vasc Surg. 1993;17:152-159.
Faught WE, Mattos MA, van Bemmelen PS, Hodgson KJ, Barkmeier LD, Ramsey DE, Sumner DS. Color-flow duplex scanning of carotid arteries: new velocity criteria based on receiver operator characteristic analysis for threshold stenoses used in the symptomatic and asymptomatic carotid trials. J Vasc Surg. 1994;19:818-828.
Neale ML, Chambers JL, Kelly AT, Connard S, Lawton MA, Roche J, Appleberg M. Reappraisal of duplex criteria to assess significant carotid stenosis with special reference to reports from the North American Symptomatic Carotid Endarterectomy Trial and the European Carotid Surgery Trial. J Vasc Surg. 1994;20:642-649.
Efron B. Bootstrap methods: another look at the jackknife. Ann Stat. 1979;7:1-26.
Kremkau FW. Doppler Ultrasound: Principles and Instruments. Philadelphia, Pa: WB Saunders Co; 1990:92-94.
Daigle RJ, Stavros AT, Lee RM. Overestimation of velocity and frequency values by multielement linear array Dopplers. J Vasc Tech. 1990;14:206-213.