This Article Abstract Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Schwartz, S. W. Articles by Howard, G. Search for Related Content PubMed PubMed Citation Articles by Schwartz, S. W. Articles by Howard, G.

(Stroke. 1997;28:343-347.)
© 1997 American Heart Association, Inc.

Articles

Consistency of Doppler Parameters in Predicting Arteriographically Confirmed Carotid Stenosis

Skai W. Schwartz, PhD; Lloyd E. Chambless, PhD; William H. Baker, MD; Joseph P. Broderick, MD; George Howard, DrPH for the Asymptomatic Carotid Atherosclerosis Study Investigators

the Department of Epidemiology and Biostatistics, College of Public Health of University of South Florida (Tampa) (S.W.S.); the Department of Biostatistics, University of North Carolina at Chapel Hill (L.E.C.); the Section of Vascular Surgery, Loyola University Medical School, Maywood, Ill (W.H.B.); the Department of Neurology of University of Cincinnati (Ohio) Medical School (J.P.B.); and the Department of Public Health Sciences and Stroke Center, Bowman Gray School of Medicine of Wake Forest University, Winston Salem, NC (G.H.).

	Abstract

Top Abstract Introduction Materials and Methods Results Discussion References

 
Background and Purpose While internal carotid peak systolic velocity (IPSV) is reportedly the best Doppler parameter for predicting lower grades of carotid artery stenosis, the internal carotid end-diastolic velocity (IEDV) or the ratio of IPSV to common carotid end-diastolic velocity (CEDV) is helpful in increasing prediction of higher grade stenoses. It is important to examine the consistency of these findings across machine and technician.

Methods Using data from 10 devices from the Asymptomatic Carotid Atherosclerosis Study, we examined the predictive ability of seven Doppler parameters: IPSV, IEDV, CEDV, common carotid peak systolic velocity (CPSV), and the ratios of IPSV/CEDV, IEDV/CEDV, and IEDV/CEDV. To assess the agreement between Doppler and arteriography in classifying percent stenosis above or below a given criterion, sensitivity, specificity, area under the receiver operating curve, and statistics were obtained from logistic models. The single best Doppler parameter for each of two grades of stenosis (60% and 80%) was determined, and its predictive ability was compared with that of IPSV. The usefulness of IEDV or IPSV/CEDV in addition to IPSV to determine higher grade stenosis was examined.

Results IPSV was the best predictor in 9 of 10 devices at 60% and in 4 devices at 80% stenosis. When another parameter was better than IPSV, the improvement was minimal. Including IEDV or IPSV/CEDV in addition to IPSV did not notably improve predictive ability.

Conclusions IPSV is the single best Doppler parameter for distinguishing severe (>80%) from less severe carotid stenosis. Information from other Doppler parameters in addition to IPSV is unlikely to be helpful.

Key Words: angiography • carotid stenosis • Doppler • ultrasonics

	Introduction

Top Abstract Introduction Materials and Methods Results Discussion References

 
It has been concluded that endarterectomy reduces the risk of incident stroke in symptomatic patients with 70% to 99% stenosis and in asymptomatic patients with 60% to 99% stenosis.^{1 2 3} The acknowledgment of an effective preventive procedure for stroke has increased the importance of the Doppler examination as a noninvasive screen for detecting clinically important carotid stenosis. Most researchers have been interested in defining the Doppler cut points that identify patients with 60% carotid stenosis, as measured by arteriography, who would benefit from carotid endarterectomy. Other researchers have sought to discriminate between severe (>80%) and moderate (60% to 80%) levels of carotid stenosis to adjust treatment comparisons for baseline degree of stenosis or to compare treatment benefits across subgroups. Thus, there is interest in using Doppler to identify persons with 80% and even 90% stenosis.

Controversy persists about the Doppler parameter that is most predictive of high-grade (>80%) carotid stenosis. Some researchers have reported that the IPSV is the single best parameter.⁴ Others have stated that IPSV in conjunction with IEDV provides the best combination for prediction.⁵ Moneta et al⁶ found that the ratio of IPSV to CPSV was the most useful parameter, and Knox et al⁷ reported that the IPSV to CEDV ratio was the most predictive parameter. Hunink et al⁴ observed that the latter ratio was useful in predicting higher grade stenosis when combined with IPSV.

Each of the above studies gave results for a single machine from a single hospital. Data from ACAS provided the opportunity to test the consistency of these results across several Doppler devices operated by different technicians.⁸ We tested the premise that IPSV was the Doppler parameter most predictive of severe stenosis for most machines. Second, we examined whether either IEDV or the IPSV/CEDV ratio was useful in predicting higher grade stenosis when combined with IPSV.

	Materials and Methods

Top Abstract Introduction Materials and Methods Results Discussion References

 
ACAS was a randomized multicenter trial that found that endarterectomy plus medical treatment was superior to medical management alone in preventing ipsilateral transient ischemic attacks and strokes over a 5-year period in asymptomatic patients with carotid artery stenosis of at least 60%. In ACAS, the Doppler examination played an important role in determining patient eligibility. Patients could be enrolled in ACAS if a Doppler examination indicated a carotid stenosis of at least 60%, although a preoperative confirmatory arteriogram was required for those patients randomized to surgery.

Carotid stenosis was measured by the ACAS and Veterans Administration Asymptomatic Trial method, sometimes referred to as the "NASCET" method.⁹ The minimal residual lumen (MRL) was compared with the distal lumen (DL) at a point distal to the carotid sinus where the walls become parallel, and a percent diameter stenosis was calculated as % Stenosis=(1-[MRL/DL])_*100. Measurements were made in a similar manner across all centers using an identical plastic ruler, and measurements were rounded to the nearest 0.5 mm. Interarterial angiography was performed by either digital or conventional "cut film" techniques. Digital examinations could be magnified; however, all measurements were obtained manually using the standard ruler (no electronic program measurements were allowed). Details of arteriographic quality control efforts are provided elsewhere.⁹

As part of the certification process for participation in ACAS, each center was required to provide data on 50 consecutive patients who had both a Doppler examination and an arteriogram within a 6-week interval. When ACAS ceased randomization, data had been received from 65 Doppler devices in 37 clinics. This phase of ACAS was referred to as the "validation study."

For the present analysis, 10 machines from nine clinics were selected from the validation study on the basis of sample size. Specifically, a machine was chosen if data were available from that machine for (1) at least 70 arteries overall and (2) at least 10 arteries with a carotid stenosis of at least 80% determined arteriographically. These criteria were imposed to allow for meaningful analysis.

Seven Doppler parameters were examined: IPSV, IEDV, CPSV, and CEDV and the ratios IPSV/CPSV, IEDV/CEDV, and IPSV/CEDV.

Statistical Analysis
Logistic regression with a forward selection procedure was used to determine the single best Doppler parameter. The best parameter is defined as the one with the greatest association to arteriographic classification (above versus below the given criterion), as measured by the standardized coefficient, and the smallest probability value when testing the null hypothesis of no association between the parameter and percent stenosis. (In the logistic model, the standardized coefficient is the logarithmically transformed odds ratio, divided by its standard error.)

Agreement statistics, namely, sensitivity, specificity, and ,¹⁰ were calculated by comparing Doppler classifications of 60% and 80% stenosis to respective arteriographic results. Doppler classifications were based on whether the logistic probability of having higher grade stenosis was less than or greater than 0.5. The area under the ROC was calculated by the method of Hanley and McNeil¹¹ using consecutive logistic probability intervals of length 0.05 as Doppler cut points.

If a parameter other than IPSV was the best parameter, its predictive ability was compared with that of IPSV by using the likelihood ratio criterion as follows: The predictive abilities of IPSV alone and the predictive ability of the best parameter alone were each compared with the predictive ability of the two parameters together. If the first comparison is statistically significant, then adding the best parameter gives a better model fit than does IPSV alone; if the second comparison is statistically significant, then adding IPSV gives a better model fit than does the best parameter alone. If neither comparison is significant, then there is no reason to prefer one over the other.

In addition, the likelihood ratio criterion was used to test whether the inclusion of the ratio or IEDV was significantly better than using IPSV alone.

Because model fit may be strongly influenced by a minority of points with high Doppler values, whereas sensitivity and specificity are not, it is possible that the inclusion of a Doppler parameter results in a significantly better model fit without impacting the discriminant ability of the machine. However, the reverse is unlikely to be true. If there is an increase in discriminant ability of the machine, then this should be apparent in the test of model fit.

	Results

Top Abstract Introduction Materials and Methods Results Discussion References

 
The 10 machines are from eight US hospitals and one Canadian hospital. (Machines 1 and 2 were used at the same hospital.) Machines ranged in technological capability from very basic continuous-wave Doppler to sophisticated imaging and color Doppler. Reported results are in velocity units except for machine 5, for which results are reported in frequency units. The percentage of arteries determined by arteriography to be >60% stenosed ranged from 27% (44/164) to 53% (79/148), and the percentage of arteries determined to have >80% stenosis ranged from 8% (10/126) to 21% (31/148) (Table 1).

View this table: [in this window] [in a new window]   Table 1. Arteries With True Percent Stenosis >60% and >80% by Arteriogram Evaluated by Each Machine

IPSV proved to be the single best Doppler predictor of arteriographic percentage stenosis >60% for 9 machines and of stenosis >80% for 4 machines (Table 2). IEDV was the best predictor of 60% stenosis for 1 machine and of 80% stenosis for 2 machines. The IPSV/CPSV ratio was the best predictor of 80% stenosis for just 2 machines.

View this table: [in this window] [in a new window]   Table 2. Summary Statistics Comparing IPSV Alone With Best Predictor Alone for Machines for Which IPSV Was Not Best Predictor at Indicated Stenosis Level

Even when the best predictor was superior to IPSV based on the likelihood ratio criterion, the degree of improvement based on and AUC statistics was slight. The machines for which IPSV was not the best predictor appeared to have lower validity than the other machines, even when the best predictor was used instead of IPSV for these machines. For example, at 80% stenosis, the average AUC for the 6 machines for which IPSV was not the best predictor was 84.6 versus 92.4 for the remaining 4 machines. This result was statistically significant (P<.05) based on a Student's t test. Corresponding statistics were 30.7% and 51.2%, respectively.

Table 3 compares at each of the two grades of stenosis the discriminant ability of three models: (1) only IPSV, (2) both IPSV and the IPSV/CEDV ratio, and (3) both IPSV and IEDV. The addition of the ratio resulted in no significant improvement in model fit at 60% stenosis, and a significantly better model fit was noted for just one of the 10 machines at 80% stenosis (machine 6). For this machine, a small increase in but no increase in AUC was noted. Although model fit was improved by the addition of IEDV for machines 3, 8, and 9, predictability based on the agreement statistics was not notably improved for any machine by this addition.

View this table: [in this window] [in a new window]   Table 3. Summary Statistics Comparing Three Models to Detect 60% and 80% Stenosis

Fig 1 depicts the ROCs for predicting 60% and 80% stenosis for the two ALT Mark 9 machines (Nos. 5 and 9) based on three models: (1) IPSV alone, (2) IPSV+IEDV, and (3) IPSV+IPSV/IEDV. Machines 5 and 9 were selected for illustration because they are both ALT Mark 9 devices, which are the most advanced machines in our series of 10 machines. Fig 1 shows a visibly lower ROC and reduced AUC in the model including both IPSV and its ratio to IEDV relative to IPSV alone for machine 5 at 80% stenosis. This indicates that it is possible to reduce the discriminant ability of the Doppler by including the ratio in the model.

View larger version (45K): [in this window] [in a new window]   Figure 1. ROCs for detection of stenosis of >60% (left) or >80% (right) based on IPSV, IPSV+IEDV, and IPSV+IPSV/CEDV ratio for ATL Mark 9 Doppler machines.

Fig 2 is a scatterplot of IPSV and IEDV values versus arteriogram percent stenosis for patients whose predicted stenosis based on IPSV is >60% for machines 5 and 9. Both IPSV and IEDV increase with increasing stenosis beyond 60%. From the graphs, it is not clear that IEDV provides any information additional to IPSV in determining higher grade stenosis.

View larger version (51K): [in this window] [in a new window]   Figure 2. IPSV and IEDV versus arteriogram stenoses for patients whose IPSV indicates >60% stenosis for ATL Mark 9 Doppler machines.

	Discussion

Top Abstract Introduction Materials and Methods Results Discussion References

 
To our knowledge, our study is the first to examine Doppler parameters across a variety of machines and technicians. Our study addresses the following questions: (1) Given an unknown Doppler device and technician, what is likely to be the single best Doppler parameter to assess stenosis? (2) Are IEDV and IPSV/CEDV likely to be helpful in improving prediction of higher grade stenosis? To the first question, the answer is IPSV, and to the second question, the answer is no.

This study confirms results seen by others^{4 5} that IPSV is the best single predictor of percent stenosis by arteriography. We were unsuccessful in finding a single Doppler parameter that was superior to IPSV in ability to predict stenosis at any stenosis level.

Our results did not support the finding by Moneta et al⁶ that the IPSV/CPSV ratio was the best parameter for a majority of machines. The systolic ratio was the best single predictor for just 2 of the 10 machines at 80% stenosis and was not the best predictor for any machine at 60% stenosis.

Because machines for which IPSV was not the best parameter had substantially lower and AUC statistics than the remaining machines, our data suggest that if another parameter is superior to IPSV in predicting stenosis, the validity of the data may be suspect. This could be due to technician or machine error. A larger study would be required to confirm this result.

We were unable to confirm the finding of Hunink et al⁴ that the ratio of the IPSV to the CEDV was helpful in predicting higher grade stenosis when included with IPSV in the model. We had the unexpected result that its inclusion was more likely to decrease predictive ability than to increase it. Apparently, including the ratio introduces variability without any compensating improvement in prediction. Likewise, we were unable to confirm results by Faught et al⁵ that using IPSV and IEDV jointly provided substantial improvement over using IPSV alone.

The machines in this study had a proven accuracy in the diagnosis of 60% stenosis based on IPSV. Machines for which IPSV was the most predictive parameter also had good accuracy at 80% stenosis. However, machines for which IPSV was not the best predictor tended to have low accuracy, and the proportionate number of such machines increased with increasing grade stenosis. On average, the accuracy of the Doppler was poor at 80% stenosis. The addition of other criteria to IPSV did not increase the accuracy of the machines in classifying stenosis above or below 80%.

Although our results are based on only 10 machines and we are limited somewhat by the small number of arteries with 80% stenosis by arteriography, we assume that they are generally applicable. Because we selected machines strictly on the basis of sample size, our selection process may favor larger hospitals and those hospitals in which there is a greater tendency to perform both a Doppler ultrasound examination and arteriography on a given patient. There may be a difference in technician accuracy between larger hospitals and smaller ones.

Also, of the 10 machines examined, only the ATLs are still available for purchase, while the others have been upgraded. However, our results appeared consistent for the state-of-the-art ATL machines included, as well as for those machines that have been discontinued. We can think of no other reasons why our findings would not be generalizable to Doppler examinations that are routinely performed in clinics today, since the addition of an image to guide placement of the sampling probe did not change the results of this study.

In ACAS, arteriographic percent stenosis was calculated using a diameter reduction and calculating the reduction relative to a distal region where the walls return to parallel. Both of these design features provide an estimate of the degree of stenosis that tends to be relatively low (that is, an 80% stenosis in ACAS would be declared as an even higher stenosis using alternative measurement techniques). Others, most notably the European Carotid Surgery Trialists, measured the diameter reduction at the site of stenosis. When the results of this study are compared with other similar studies, the method of measurements should likewise be compared. The Doppler criteria provided in this report should be interpreted as being predictive of "moderately severe" stenosis in the case of 60% arteriographic stenosis and of "severe" stenosis in the case of 80% stenosis. The relationship between Doppler and milder levels of stenoses is not addressed in this report.

Several previous reports have considered and adjusted for verification bias. In our study, the degree of verification bias likely differs among clinics. We did not have clinic information to assess this bias. On the basis of the high proportion of patients with arteriographic percent stenoses below 60% and the low proportion with stenoses above 80%, it is likely that the verification bias in our study was lower than that reported in other studies. However, our purpose was not to report Doppler cut points or sensitivity and specificity for particular machines but to determine the best Doppler parameters. Unless verification bias has a differential impact on IPSV and IEDV, it is not relevant to our analysis.

To the extent that our data are generalizable, they suggest that clinicians who base their treatment options on only the Doppler diagnosis of a >80% stenosis need to critically assess this practice. Individual laboratories may use published data as a rough guide for Doppler criteria only. Because diagnostic criteria may vary across institution and machine, each hospital needs to maintain a database comparing the Doppler and arteriographic assessment of stenosis. Besides the obvious clinical value, the collection of such data would allow for a definitive study.

	Selected Abbreviations and Acronyms

 

ACAS = Asymptomatic Carotid Atherosclerosis Study AUC = area under the curve CEDV = common carotid end-diastolic velocity CPSV = common carotid peak systolic velocity IEDV = internal carotid end-diastolic velocity IPSV = internal carotid peak systolic velocity ROC = receiver operator curve

	Acknowledgments

 
This study was supported by grant R01-NS22611 from the US Public Health Service National Institute of Neurological Disorders and Stroke. The authors acknowledge the support of all participants in the Asymptomatic Carotid Atherosclerosis Study. The authors gratefully acknowledge the editorial assistance of Calvin B. Ernst, MD (Division Head, Division of Vascular Surgery, Henry Ford Hospital) and Harold P. Adams (Professor and Director, Division of Cerebral Vascular Diseases, University of Iowa). We also appreciate the editorial assistance of Debra H. Weiner of the ACAS Operations Center.

	Footnotes

 
Reprint requests to ACAS Operations Center, Stroke Research Center, Medical Center Blvd, Winston-Salem, NC 27157-1078. E-mail vjhoward@bgsm.edu.

This article has been approved by the Executive and Publications Committees of ACAS.

Received August 22, 1996; revision received October 29, 1996; accepted October 29, 1996.

	References

Top Abstract Introduction Materials and Methods Results Discussion References

 
1. North American Symptomatic Carotid Endarterectomy Trial Collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients with high grade carotid stenosis. N Engl J Med. 1991;325:445-453.[Abstract]

2. European Carotid Surgery Trialists' Collaborative Group. MRC European Carotid Surgery Trial: interim results for symptomatic patients with severe (70-99%) or with mild (0-29%) carotid stenosis. Lancet. 1991;337:1235-1243.[Medline] [Order article via Infotrieve]

3. Executive Committee for the Asymptomatic Carotid Atherosclerosis Study. Endarterectomy for asymptomatic carotid artery stenosis. JAMA. 1995;273:1421-1428.[Abstract/Free Full Text]

4. Hunink MGM, Polak JF, Barlan MM, O'Leary DH. Detection and quantification of carotid artery stenosis: efficacy of various Doppler velocity parameters. AJR Am J Radiol. 1993;160:619-625.[Abstract/Free Full Text]

5. Faught WE, Mattos MA, van Bemmelen PS, Hodgson KJ, Barkmeier LD, Ramsey DE, Sumner DS. Color-flow duplex scanning of carotid arteries: new velocity criteria based on receiver operator characteristic analysis for threshold stenoses used in the symptomatic and asymptomatic carotid trials. J Vasc Surg. 1994;19:818-827.[Medline] [Order article via Infotrieve]

6. Moneta GL, Edwards JM, Chitwood RW, Taylor LM Jr, Lee RW, Cummings CA, Porter JM. Correlation of North American Symptomatic Carotid Endarterectomy Trial (NASCET): angiographic definition of 70% to 99% internal carotid artery stenosis with duplex scanning. J Vasc Surg. 1993;17:152-159.[Medline] [Order article via Infotrieve]

7. Knox RA, Breslau PJ, Strandness DE Jr. A simple parameter for accurate detection of severe carotid disease. Br J Surg. 1982;69:230-233.[Medline] [Order article via Infotrieve]

8. Howard G, Chambless LE, Baker WH, Riccota JJ, Jones AM, O'Leary D, Howard VJ, Elliot TJ, Lefkowitz DS, Toole JF. A multicenter validation study of Doppler ultrasound versus angiography. J Stroke Cerebrovasc Dis. 1991;1:166-173.

9. Dean BL, Lefkowitz DS, Howard VJ, Frey JF, Schwartz S, Chambless LE, Heiserman JE, Feinberg WM, Toole JF. Comparison of centralized versus `site-based' measurement of angiographic stenosis for eligibility in the Asymptomatic Carotid Atherosclerosis Trial. Invest Radiol. 1996;31:446-450.[Medline] [Order article via Infotrieve]

10. Fleiss JL. Statistical Methods for Rates and Proportions. New York, NY: John Wiley & Sons Inc; 1981:217-222.

11. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29-36.[Abstract/Free Full Text]

This article has been cited by other articles:

				C. H. Wierks and N. Labropoulos Noninvasive Carotid Imaging Perspectives in Vascular Surgery and Endovascular Therapy, June 1, 2004; 16(2): 89 - 99. [Abstract] [PDF]

				M. Artieda, A. Cenarro, A. Ganan, I. Jerico, C. Gonzalvo, J. M. Casado, I. Vitoria, J. Puzo, M. Pocovi, and F. Civeira Serum Chitotriosidase Activity Is Increased in Subjects With Atherosclerosis Disease Arterioscler Thromb Vasc Biol, September 1, 2003; 23(9): 1645 - 1652. [Abstract] [Full Text] [PDF]

				P. J. Nederkoorn, Y. van der Graaf, and M.G. M. Hunink Duplex Ultrasound and Magnetic Resonance Angiography Compared With Digital Subtraction Angiography in Carotid Artery Stenosis: A Systematic Review Stroke, May 1, 2003; 34(5): 1324 - 1331. [Abstract] [Full Text] [PDF]

				G. B. Anderson, R. Ashforth, D. E. Steinke, R. Ferdinandy, and J. M. Findlay CT Angiography for the Detection and Characterization of Carotid Artery Bifurcation Disease Stroke, September 1, 2000; 31(9): 2168 - 2174. [Abstract] [Full Text] [PDF]

				V. S. Lee, B. S. Hertzberg, M. J. Workman, T. P. Smith, M. A. Kliewer, D. M. DeLong, and B. A. Carroll Variability of Doppler US Measurements along the Common Carotid Artery: Effects on Estimates of Internal Carotid Arterial Stenosis in Patients with Angiographically Proved Disease Radiology, February 1, 2000; 214(2): 387 - 392. [Abstract] [Full Text]

				D. W. Droste, R. Jurgens, D. G. Nabavi, G. Schuierer, S. Weber, and E. B. Ringelstein Echocontrast-Enhanced Ultrasound of Extracranial Internal Carotid Artery High-Grade Stenosis and Occlusion Stroke, November 1, 1999; 30(11): 2302 - 2306. [Abstract] [Full Text] [PDF]

				V. S. Lee, B. S. Hertzberg, M. A. Kliewer, and B. A. Carroll Assessment of Stenosis: Implications of Variability of Doppler Measurements in Normal-appearing Carotid Arteries Radiology, August 1, 1999; 212(2): 493 - 498. [Abstract] [Full Text]

				J. E. Castaldo Is Carotid Endarterectomy Appropiate for Asymptomatic Stenosis?: Yes Arch Neurol, July 1, 1999; 56(7): 877 - 879. [Full Text] [PDF]

				S. Sharma, G. C. Brown, J. L. Pater, and A. F. Cruess Does a Visible Retinal Embolus Increase the Likelihood of Hemodynamically Significant Carotid Artery Stenosis in Patients With Acute Retinal Arterial Occlusion? Arch Ophthalmol, December 1, 1998; 116(12): 1602 - 1606. [Abstract] [Full Text] [PDF]

				W. T. Longstreth Jr, L. Shemanski, D. Lefkowitz, D. H. O'Leary, J. F. Polak, and S. K. Wolfson Jr Asymptomatic Internal Carotid Artery Stenosis Defined by Ultrasound and the Risk of Subsequent Stroke in the Elderly : The Cardiovascular Health Study Stroke, November 1, 1998; 29(11): 2371 - 2376. [Abstract] [Full Text] [PDF]

				W. S. Moore, R. F. Kempczinski, J. J. Nelson, and J. F. Toole Recurrent Carotid Stenosis : Results of the Asymptomatic Carotid Atherosclerosis Study Stroke, October 1, 1998; 29(10): 2018 - 2025. [Abstract] [Full Text] [PDF]

This Article Abstract Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Schwartz, S. W. Articles by Howard, G. Search for Related Content PubMed PubMed Citation Articles by Schwartz, S. W. Articles by Howard, G.