Changes Over Time in Optimal Duplex Threshold for the Identification of Patients Eligible for Carotid Endarterectomy
Background and Purpose—Two surgical trials established that carotid endarterectomy is beneficial to symptomatic patients who have a severe internal carotid artery (ICA) stenosis on angiograms. Duplex ultrasonography–derived hemodynamic parameters show a good correlation with angiography and are often used for detecting severe ICA stenoses. However, duplex performance is ultrasound machine and operator dependent. Over time both may change, possibly affecting duplex performance. We compared duplex performance of 2 time periods in 1 specific vascular laboratory using angiography as the gold standard.
Methods—Consecutive patients who underwent both angiography and duplex examinations of the ICA were evaluated (first period, 60 patients; second period, 61 patients). Peak systolic velocity and several other hemodynamic parameters and ratios were analyzed by receiver operating characteristic curves in their ability to detect severe ICA stenoses. The optimal parameter and threshold were determined for each period. Subsequently, duplex test characteristics were compared after the optimal thresholds of both the first and the second periods were applied in the second period.
Results—In both periods peak systolic velocity of the ICA was the best test parameter; areas under the receiver operating characteristic curve were similar (0.957 and 0.954, respectively). However, the optimal threshold was different. The optimal threshold in the second period was 270 cm/s. When the optimal threshold of 210 cm/s of the first period was applied in the second period, test characteristics changed significantly. Sensitivity increased from 98% to 100%, and specificity decreased from 85% to 71% (P=0.004).
Conclusions—The optimal threshold for detecting severe ICA stenoses with duplex ultrasonography in our laboratory changed over time. Individual laboratories should assess duplex accuracy regularly and adjust adopted criteria if necessary to keep diagnostic performance optimal.
The North American Symptomatic Carotid Endarterectomy Trial (NASCET) and the European Carotid Surgery Trial both established that carotid endarterectomy is beneficial to patients with symptoms of cerebral ischemia and a severe internal carotid artery (ICA) stenosis on selective angiograms (70% to 99%).1 2
In the diagnostic process of patients with carotid disease, duplex ultrasonography plays an important role because it is the initial, and in some clinics the only, diagnostic test for assessing and quantifying carotid artery disease.3 Duplex offers several hemodynamic parameters for describing blood flow across stenoses. A parameter extensively studied for grading ICA stenosis is peak systolic velocity (PSV). A good correlation between PSV values and angiographic findings has been reported in many studies.4 5 6 7 However, several other parameters have been proposed as well, including end-diastolic velocity (EDV), the ratio PSVICA/PSV common carotid artery (CCA), and the ratio EDVICA/EDVCCA.6 7 8 9 10 11
Duplex performance depends on the ultrasound machine used and the skills of the operator (ie, vascular technologists).12 13 14 15 Every individual laboratory should therefore validate its own criteria for grading ICA stenosis before duplex can be considered a reliable diagnostic tool for patient selection. Over time, equipment and vascular technologists may change, possibly affecting duplex performance.
Our hypothesis was that for duplex to remain a reliable diagnostic tool for patients with carotid disease, locally adopted criteria might need adjustment after a period of time. Therefore, we studied the value of different duplex-derived hemodynamic parameters and ratios for detecting severe ICA stenoses in 2 time periods and compared the diagnostic performance of duplex in these periods.
Subjects and Methods
To obtain duplex criteria for detecting a stenosis of ≥70%, we evaluated the measurements of 60 consecutive patients (119 bifurcations) who underwent both intra-arterial digital subtraction angiography (IA-DSA) and duplex examinations between July 1992 and February 1994 (the first period). The population consisted of 41 men and 19 women. The mean age was 64 years (range, 40 to 81 years). The mean interval between duplex and IA-DSA was 25 days (range, 0 to 113 days).16 To reevaluate the adopted duplex criteria, we repeated the study in 61 consecutive patients (120 bifurcations) who underwent both IA-DSA and duplex examinations between October 1996 and August 1997 (the second period). This population consisted of 47 men and 14 women. The mean age was 65 years (range, 44 to 81 years). The mean interval between duplex and IA-DSA was 17 days (range, 0 to 91 days).
Indications for referral for IA-DSA were similar in both periods. Patients with symptoms of carotid disease in the past 6 months, including transient ischemic attack, stroke, and amaurosis fugax, were first screened by duplex. Patients with PSV values in the symptomatic ICA of ≥150 cm/s (suspect for carotid disease) or with a suspected subtotal stenosis or occlusion were subsequently referred for IA-DSA.
Five different vascular technicians performed the duplex examinations. However, in 6 years 2 vascular technicians left and were replaced by new trainee technicians, whereas the 3 other vascular technicians became more experienced in performing duplex examinations.
Duplex was performed with an ATL Ultramark 9 HDI (Advanced Technology Laboratories). A sample volume of 1.5 mm was used, and the Doppler angle was aligned to the jet and kept below 60 degrees. In the first period a 10–5 MHz broadband linear array transducer with a pulsed Doppler frequency of 5 MHz was used. For more deeply situated carotid bifurcations, a 5-MHz linear array transducer with a pulsed Doppler frequency of 4 MHz was used. Conversely, in the second period a 7–4 MHz broadband linear array transducer with a pulsed Doppler frequency of 4 MHz was used in all patients to visualize the carotid bifurcation.
The technique for performing duplex was consistent in both periods. The pulsed Doppler gate was positioned in the center of the CCA ≈2 cm proximal to the carotid bifurcation, and a spectral waveform was obtained. From this spectrum, PSVCCA and EDVCCA were derived. Subsequently, the area with the most severe stenosis was located with color Doppler, and a Doppler spectral waveform was obtained at the point of the greatest mean frequency shift. From this spectrum, PSVICA and EDVICA were derived.
Intra-Arterial Digital Subtraction Angiography
The carotid bifurcation was imaged in ≥2 different directions (lateral, posteroanterior, or oblique). The view showing the most severe stenosis was used for comparison with duplex. In the first period, lumen reduction measurements were performed with the NASCET method (Stenosis=[1−(Minimal Residual Lumen/Distal Normal ICA Lumen Diameter)]×100%) on printed hard copies with a ×7 magnifying loupe marked in 0.1-mm increments. In the second period, measurements were performed with the NASCET method on hard copies by another observer using a caliper with a digital display (PAV Electronic; resolution, 0.01 mm). To exclude that variability due to different observers or methods of measuring stenosis on angiograms would influence the analysis of duplex performances, the observer from the second period repeated the measurements on the hard copies from the first period using the caliper with the digital display.
Duplex and IA-DSA were interpreted independently from each other. IA-DSA was considered the gold standard for grading carotid stenosis.
We evaluated PSVICA, EDVICA, the ratio PSVICA/PSVCCA, and the ratio EDVICA/EDVCCA by receiver operating characteristic (ROC) analysis in their ability to detect an ICA stenosis of ≥70%.17 Occluded ICA on IA-DSA, as well as ICA showing no velocity or very low velocities with high resistance signals and minimal residual lumen, associated with occlusion and subtotal stenosis, respectively, were excluded from ROC analysis.
To assess the best hemodynamic parameter or ratio for detecting an ICA stenosis of ≥70%, the areas under the curve (AUC) and corresponding SEs were calculated and compared. The parameter or ratio with the largest AUC was defined as the best test criterion. Because we used duplex as a screening tool for carotid disease, the threshold value was optimized by maximizing sensitivity while maintaining high test accuracy for both periods independently.18
In addition, logistic regression analysis was used in both periods to assess whether combining the various hemodynamic parameters and ratios would increase duplex performance (ie, increase the AUC). Models were fit with various combinations of single parameters and ratios to estimate regression coefficients. Next, data of individual patients were placed in the regression equation for estimation of the probability of an ICA stenosis of ≥70%. These estimated probabilities were then subjected to ROC analysis.
The McNemar test, a nonparametric test for 2 related dichotomous variables, was used to compare the test characteristics of duplex that would have been obtained if the optimal threshold of the first period had been applied in the second period instead of its own optimal threshold.
Table 1⇓ shows the IA-DSA–derived ICA lumen reduction measurements categorized as no stenosis to mild stenosis (0% to 29%), moderate stenosis (30% to 69%), severe stenosis (70% to 99%), and occlusion (100%) for both periods. From the 15 occlusions or suspected occlusions diagnosed with duplex in the first period, 13 were confirmed with IA-DSA and 2 were diagnosed as subtotal stenosis, whereas 3 occlusions on IA-DSA were diagnosed as severe stenosis with duplex. For further ROC analysis, PSVICA values were available for 101 arteries, EDVICA values for 92, PSVICA/PSVCCA ratios for 101, and EDVICA/EDVCCA ratios for 92. In the second period, 1 subtotal stenosis and 14 occlusions or suspected occlusions diagnosed with duplex were all confirmed with IA-DSA, whereas 1 occlusion on IA-DSA was diagnosed as a subtotal stenosis with duplex. For further ROC analysis, PSVICA values were available for 104 arteries, EDVICA values for 100, PSVICA/PSVCCA ratios for 103, and EDVICA/EDVCCA ratios for 98.
Table 2⇓ shows the results of the ROC analyses for the different hemodynamic parameters and ratios from the first and the second periods expressed as AUC. In the first period, both parameter PSVICA and ratio PSVICA/PSVCCA were the most accurate, with an AUC of 0.957 and 0.959, respectively. We defined PSVICA as the best test parameter for detecting a ≥70% stenosis of the ICA because it was the easiest to obtain and did not require calculations. In the second period, parameter PSVICA was the most accurate with an AUC of 0.954. Table 3⇓ shows the results of logistic regression analysis, which demonstrated that combining the various parameters and ratios did not significantly increase duplex performance in both the first and the second periods.
The Figure⇓ shows the ROC curves associated with the ROC analyses of PSVICA with different threshold values for a stenosis of ≥70%. In the ROC analysis of data from the first period, an optimal PSVICA threshold of 210 cm/s was observed (sensitivity of 100% [95% CI, 88% to 100%]; specificity of 84% [95% CI, 73% to 92%]).16 When the angiograms from the first period were analyzed by the observer from the second period, the AUC and optimal PSVICA threshold (0.952 and 200 cm/s, respectively) were similar to the original analysis. In the second period, however, a PSVICA of 270 cm/s appeared to be the optimal threshold for a stenosis of ≥70% (sensitivity of 98% [95% CI, 87% to 99%]; specificity of 85% [95% CI, 74% to 92%]). Applying the optimal PSVICA threshold of 210 cm/s from the first period in the second period instead of using the threshold of 270 cm/s would significantly affect the test characteristics (P=0.004). Sensitivity would increase to 100%, and specificity would decrease to 71%.
Our study showed that even when only 1 type of ultrasound machine is used in a specific vascular laboratory, locally adopted duplex criteria for detecting a ≥70% ICA stenosis do not remain valid over time. Of the different acquired hemodynamic parameters and ratios, PSVICA was considered the best test parameter in the first as well as in the second period, but its optimal threshold was not consistent.
ROC curves facilitate the choice of a threshold. If duplex is used as a screening tool, a high sensitivity is required in order not to miss any patient who may benefit from carotid endarterectomy. On the other hand, if duplex is used as the sole preoperative test for carotid disease, a high specificity is required as well in order not to perform unnecessary carotid endarterectomy. Because we use duplex as a screening tool for carotid disease, we chose a PSVICA threshold of 210 cm/s on the basis of its high sensitivity, while maintaining high test accuracy, in the first period. Conversely, nearly the same sensitivity and specificity were achieved with a threshold of 270 cm/s in the second period. Differences in the method of stenosis measurement on angiograms between the 2 observers could not explain the change over time of the optimal PSVICA threshold. When we applied the optimal threshold of 210 cm/s of the first period in the second period, the number of false-positive duplex examinations increased remarkably, which would have led to significantly more carotid IA-DSA procedures, with a reported risk of stroke or death ranging from 1% to 4% in patients with atherosclerosis.19 20
A range of PSVICA thresholds has been suggested for detecting an ICA stenosis of ≥70% according to NASCET criteria. Carpenter et al4 proposed a PSVICA threshold of 210 cm/s, whereas Neale et al5 found a PSVICA of 270 cm/s to be the optimal threshold with high sensitivity and high accuracy. The wide variability in published criteria indicates the importance of individual validation of duplex criteria against the gold standard of IA-DSA before duplex can be considered a reliable diagnostic tool for patient selection. However, we found that even when individual duplex criteria have been established, these may not remain valid for a long time.
The source of the variability in thresholds that we found is not fully understood. Possible explanations are the replacement of 2 of the 5 vascular technologists, the differences in the technologists’ technique, and the changing level of experience.12 14 15 Additionally, changes within the ultrasound machine (ie, upgrades) can contribute to the variability in thresholds.21 22 23 24 Two major changes occurred in addition to several other software upgrades in the past years. The 7–4 MHz broadband linear array transducer became available in between the first period and the second period (1995), which improved the visualization of the carotid bifurcation remarkably and hence facilitated the acquisition of the PSVICA at the point of the greatest mean frequency shift. Furthermore, real-time Doppler analysis was introduced in the middle of the first period (1993), which allowed vascular technologists to automatically trace the Doppler waveforms and distillate the highest PSV and EDV values.
Variability in diagnostic duplex criteria across vascular laboratories and ultrasound machines has been pointed out by others.13 15 Kuntz et al13 compared 2 vascular laboratories in their ability to detect an ICA stenosis of ≥70% using various duplex-derived hemodynamic parameters and ratios. They found practically no differences in AUC, but the optimal threshold value for each parameter or ratio was different for each laboratory (eg, the PSVICA threshold for one laboratory was 229 cm/s and for the other 340 cm/s). However, they optimized the thresholds both by maximizing accuracy and by minimizing the 2-year risk of stroke in case duplex would be used as the sole preoperative test for carotid disease. This approach led to different tradeoffs in sensitivity and specificity for each laboratory, which by itself can be a reason for different optimal thresholds for each laboratory (eg, for PSVICA, sensitivity and specificity for one laboratory were 95.0 and 82.5, respectively; for the other laboratory, the values were 89.5 and 93.2, respectively). Their main explanation for the disparity between laboratories was the differences in ultrasound machines used. In contrast, Alexandrov et al15 compared the optimal threshold values of 2 laboratories that used similar equipment. They found major differences in sensitivity and specificity when the PSVICA threshold of one laboratory was applied in the other laboratory instead of this laboratory using its own previously validated criterion. Because the ultrasound machines were similar, their explanation for the difference in diagnostic duplex criteria was the difference in the technologists’ technique.
Grading of carotid stenosis is essential for determining the risk of stroke and subsequent patient treatment. Changes over time in duplex criteria for detecting a ≥70% ICA stenosis constitute a profound problem, particularly for those centers that select patients for carotid endarterectomy with duplex alone. Because duplex performance is operator and ultrasound machine dependent, neither should be changed after acceptable duplex accuracy and validation of duplex criteria are achieved. Before duplex criteria are validated, it should be ascertained that the operators are experienced and expect to be working for a long period of time in the vascular laboratory. In addition, proposed adjustments of the validated ultrasound machines by the manufacturer should be evaluated carefully before they are implicated.
Our study demonstrated that duplex criteria may change over time. However, we did not analyze the possible causes. Therefore, it is difficult to indicate a time interval or event after which duplex criteria should be validated again. Nevertheless, in our case the optimal duplex threshold changed significantly after a time interval of 2.5 years, during which operators and the ultrasound machine gradually changed. We recommend that individual vascular laboratories continuously compare their duplex results with the results of other correlating methods, such as IA-DSA, MR angiography, and CT angiography, either performed in their center or in another center, to assess duplex accuracy. Furthermore, patients should be referred for IA-DSA when the duplex result of the symptomatic ICA is inconclusive, thus providing a gold standard for duplex results of both ICAs.
- Received April 3, 1998.
- Revision received August 3, 1998.
- Accepted August 12, 1998.
- Copyright © 1998 by American Heart Association
Chaturvedi S, Policherla PN, Femino L. Cerebral angiography practices at US teaching hospitals. Stroke. 1997;28:1895–1897.
Carpenter JP, Lexa FJ, Davis JT. Determination of duplex Doppler ultrasound criteria appropriate to the North American Symptomatic Carotid Endarterectomy Trial. Stroke. 1996;27:695–699.
Neale ML, Chambers JL, Kelly AT, Connard S, Lawton MA, Roche J, Appleberg M. Reappraisal of duplex criteria to assess significant carotid stenosis with special reference to reports from the North American Symptomatic Carotid Endarterectomy Trial and the European Carotid Surgery Trial. J Vasc Surg. 1994;20:642–649.
Moneta GL, Edwards JM, Chitwood RW, Taylor LMJ, Lee RW, Cummings CA, Porter JM. Correlation of North American Symptomatic Carotid Endarterectomy Trial (NASCET) angiographic definition of 70% to 99% internal carotid artery stenosis with duplex scanning. J Vasc Surg. 1993;17:152–157.
Chang YJ, Golby AJ, Albers GW. Detection of carotid stenosis: from NASCET results to clinical practice. Stroke. 1995;26:1325–1328.
Alexandrov AV, Brodie DS, McLean A, Hamilton P, Murphy J, Burns PN. Correlation of peak systolic velocity and angiographic measurement of carotid stenosis revisited. Stroke. 1997;28:339–342.
Kuntz KM, Polak JF, Whittemore AD, Skillman JJ, Kent KC. Duplex ultrasound criteria for the identification of carotid stenosis should be laboratory specific. Stroke. 1997;28:597–602.
Howard G, Baker WH, Chambless LE, Howard VJ, Jones AM, Toole JF, for the Asymptomatic Carotid Atherosclerosis Study Investigators. An approach for the use of Doppler ultrasound as a screening tool for hemodynamically significant stenosis (despite heterogeneity of Doppler performance): a multicenter experience. Stroke. 1996;27:1951–1957.
Alexandrov AV, Vital D, Brodie DS, Hamilton P, Grotta JC. Grading carotid stenosis with ultrasound: an interlaboratory comparison. Stroke. 1997;28:1208–1210.
van Leersum M, van Leeuwen MS, van der Schouw Y, Mali WPTM. Duplex threshold values for identification of carotid stenosis greater than 70. Radiology. 1994;193:298. Abstract.
Wilterdink JL, Feldmann E, Easton JD, Ward R. Performance of carotid ultrasound in evaluating candidates for carotid endarterectomy is optimized by an approach based on clinical outcome rather than accuracy. Stroke. 1996;27:1094–1098.
Hankey GJ, Warlow CP, Sellar RJ. Cerebral angiographic risk in mild cerebrovascular disease. Stroke. 1990;21:209–222.
Davies KN, Humphrey PR. Complications of cerebral angiography in patients with symptomatic carotid territory ischaemia screened by carotid ultrasound. J Neurol Neurosurg Psychiatry. 1993;56:967–972.
Nelson TR, Pretorius DH. Device for the calibration of flow-velocity-measuring Doppler ultrasound equipment. J Ultrasound Med. 1990;9:575–581.
Daigle RJ, Stavros AT, Lee RM. Overestimation of velocity and frequency values by multielement linear array Dopplers. J Vasc Technol. 1990;14:206–213.