A Risk Factor-based Predictive Model of Outcomes in Carotid Endarterectomy
The National Surgical Quality Improvement Program 2005–2010
Background and Purpose—Accurate knowledge of individualized risks and benefits is crucial to the surgical management of patients undergoing carotid endarterectomy (CEA). Although large randomized trials have determined specific cutoffs for the degree of stenosis, precise delineation of patient-level risks remains a topic of debate, especially in real world practice. We attempted to create a risk factor-based predictive model of outcomes in CEA.
Methods—We performed a retrospective cohort study involving patients who underwent CEAs from 2005 to 2010 and were registered in the American College of Surgeons National Quality Improvement Project database.
Results—Of the 35 698 patients, 20 015 were asymptomatic (56.1%) and 15 683 were symptomatic (43.9%). These patients demonstrated a 1.64% risk of stroke, 0.69% risk of myocardial infarction, and 0.75% risk of death within 30 days after CEA. Multivariate analysis demonstrated that increasing age, male sex, history of chronic obstructive pulmonary disease, myocardial infarction, angina, congestive heart failure, peripheral vascular disease, previous stroke or transient ischemic attack, and dialysis were independent risk factors associated with an increased risk of the combined outcome of postoperative stroke, myocardial infarction, or death. A validated model for outcome prediction based on individual patient characteristics was developed. There was a steep effect of age on the risk of myocardial infarction and death.
Conclusions—This national study confirms that that risks of CEA vary dramatically based on patient-level characteristics. Because of limited discrimination, it cannot be used for individual patient risk assessment. However, it can be used as a baseline for improvement and development of more accurate predictive models based on other databases or prospective studies
Decision making regarding the need for carotid intervention is heavily dependent on whether the postoperative risk of stroke, myocardial infarction, or death outweighs the risks of conservative management. Since the publication of large randomized, controlled trials (RCTs), 1–5 carotid endarterectomy (CEA) has been favored over conservative management in a select group of patients. This benefit is contingent on a favorable operative risk profile of the patients. However, the rigorous selection criteria of RCTs restrict their results in certain patients with low operative risk and predefined age range. The comparison of an estimated risk of adverse events for each individual patient with these benchmarks could tailor the application of the results of evidence-based medicine. This strategy can also allow for the identification of modifiable risk factors associated with stroke, myocardial infarction (MI), and death in patients undergoing CEA.
There have been several studies attempting to identify such modifiable risk factors. Most of these have been retrospective analyses of single institution experiences,6–8 demonstrating results with limited generalization given their inherent selection bias. The interpretation of other systematic reviews or multicenter studies9–16 is equally limited given their focus on regional or subgroup data and administrative registries. The latter have been heavily criticized for not being independently validated and underreporting patient comorbidities and procedural complications.
These limitations are at least partially addressed by the American College of Surgeons (ACS) National Quality Improvement Program (NSQIP) database, which contains prospectively collected data from >180 private and academic hospitals across the country. It allows for the unrestricted study of the patient population in question through high quality and reliable data sets.17 Using this database, preoperative comorbidities associated with postoperative stroke, MI, death, or their combination in patients undergoing CEA were identified. Based on these data, a risk factor-based predictive model of negative outcomes in CEA was developed.
All of the patients undergoing CEA in the ACS NSQIP database between 2005 and 2010 were included in the analysis. The ACS NSQIP prospectively collects data on >200 variables pertaining to patient characteristics, comorbid conditions, operative details, and 30-day postoperative outcomes for a variety of surgical procedures. Data for the NSQIP database are collected in each fully participating site by a surgical clinical reviewer, who is typically a trained nurse. To ensure that the data collected are of the highest quality, ACS has developed a host of different training mechanisms for the surgical clinical reviewers and conducts an interrater reliability audit of select participating sites. An interrater reliability audit disagreement rate of >5% or 30-day follow-up rate <80% results in an additional audit and exclusion of the site from the calculations. Preoperative, intraoperative, and postoperative data (until 30 days after the procedure) are collected for all of the patients. The latest interrater reliability audit for participating sites has revealed an overall disagreement rate of 1.99%. More information about ACS NSQIP, including diagnostic criteria for the risk factors included in this analysis, is available at http://www.acsnsqip.org.
To establish a cohort of patients undergoing CEA for asymptomatic stenosis, we used current procedural terminology code 35301 to identify patients in the registry who underwent CEA between 2005 and 2010. Symptomatic status was then assigned to any subject with a history of cerebrovascular accident with or without neurological deficit or a history of transient ischemic attack (TIA). The NSQIP database does not provide details on the side of the stroke or ischemic attack and, therefore, subjects were classified as symptomatic regardless of whether the symptoms were in the distribution of the operated artery. All of the remaining cases were defined as asymptomatic.
The primary outcome variable was the 30-day postoperative risk of stroke, MI, death, or their combination for patients registered in NSQIP-ACS undergoing CEA. The effect on the CEA outcomes of the pertinent exposure variables was examined in a multivariate analysis including all of the variables (19 variables). The variables were continuous for age and body mass index. Categorical variables were sex, chronic obstructive pulmonary disease (COPD), peripheral vascular disease, history of MI, angina, TIA, stroke with and without residual symptoms, congestive heart failure, alcohol consumption (>2 drinks per day in the 2 weeks before admission), previous coronary angioplasty, previous coronary artery bypass grafting, smoking (patient having smoked cigarettes in the year before admission for surgery), hypertension (persistent elevation of systolic blood pressure >140 mm Hg or a diastolic pressure >90 mm Hg or requires an antihypertensive treatment at the time that the patient is being considered as a candidate for surgery), bleeding disorder (any condition that places the patient at risk for excessive bleeding requiring hospitalization because of a deficiency of blood clotting elements), diabetes mellitus, and dialysis. Year of procedure and hospital size (not available in NSQIP) were not included in the model because the focus was on patient-level factors.
The multivariate logistic regression parameter values were used to construct a predictive model for postoperative complications. Logistic regression was used to develop a multivariable model for the prediction of the combined outcome. The C-index (area under the receiver operating characteristic curve) was used to measure discriminatory ability. A value of 0.5 indicates a model of no discriminatory ability at all (not better than chance). A C-index corrected for overfitting bias was calculated using leave-out cross-validation (ie, leave 4% out or ≈40 events and total of 1427 patients, repeated 1000 times, as well as bootstrap validation, repeated 1000 times). The Hosmer-Lemeshow test was used to assess calibration of the model. Interactions were tested but none were significant at the threshold (0.001) that we set to correct for multiple testing (20*19/2=190 interactions in all), and, further, none improved the C-index by >0.002. Nonlinear functions of the continuous characteristics (age and body mass index) were explored using regression splines, but none improved the model. Statistical analyses were performed using the XLSTAT version 2011.2.01 (Addinsoft) and R (version 2.15.1).
Demographics and Clinical Characteristics
In the selected study period there were 35 698 patients (mean age was 71.1 years, with 59.1% men) registered in NSQIP, of whom 20 015 were asymptomatic (56.1%) and 15 683 symptomatic (43.9%). The 2 groups demonstrated significant differences in the prevalence of the following risk factors: sex, COPD, history of peripheral vascular disease, congestive heart failure, history of percutaneous coronary angioplasty, coronary artery bypass grafting, history of bleeding disorders, and increased body mass index (Table 1).
The 30-day incidences of stroke, MI, death, or their combined after CEA were 1.64%, 0.69%, 0.75%, and 2.78%, respectively (Table 2). Symptomatic patients demonstrated a significantly higher incidence of stroke and death but not MI after CEA.
Multivariate Analysis of 30-Day Postoperative Outcomes
Previous stroke was independently (Figure 1) associated with a higher risk of postoperative stroke within 30 days (odds ratio [OR], 2.55 [95% CI, 2.05-3.17]), death (OR, 2.98 [95% CI, 2.18-4.06]), and the composite outcome (OR, 2.26 [95% CI, 1.90-2.69]). In addition to having a previous stroke or TIA, the only other variable independently associated with a higher risk of postoperative stroke was history of angina (OR, 1.89 [95% CI, 1.19-2.99]).
Postoperative MI within 30 days (Figure 1) was independently associated with older age, previous history of angina (OR, 3.50 [95% CI, 2.07-5.93]), coronary angioplasty (OR, 1.59 [95% CI, 1.13-2.24]), stroke (OR, 1.59 [95% CI, 1.10-2.30]), and diabetes mellitus (OR, 1.57 [95% CI, 1.14-2.16]). No such association was identified with previous MI or congestive heart failure.
Risk factors independently associated (Figure 1) with a higher risk of postoperative death within 30 days were: older age (OR, 1.05 [95% CI, 1.04-1.07]), history of COPD (OR, 2.78 [95% CI, 1.97-3.91]), MI (OR, 2.41 [95% CI, 1.25-4.66]), congestive heart failure (OR, 2.94 [95% CI, 1.52-5.68]), peripheral vascular disease (OR, 1.70 [95% CI, 1.14-2.53]), stroke (OR, 2.98 [95% CI, 2.18-4.06]), and dialysis (OR, 3.72 [95% CI, 1.85-7.45]).
Combined Stroke, MI, or Death
The following risk factors were independently associated (Figure 1) with a higher risk of the composite outcome within 30 days: older age (OR, 1.02 [95% CI, 1.01-1.03]), sex (OR, 1.18 [95% CI, 1.01-1.37]), history of COPD (OR, 1.44 [95% CI, 1.17-1.78]), MI (OR, 1.59 [95% CI, 1.04-2.45]), angina (OR, 2.29 [95% CI, 1.66-3.16]), congestive heart failure (OR, 2.09 [95% CI, 1.32-3.31]), peripheral vascular disease (OR, 1.33 [95% CI, 1.06-1.67]), stroke (OR, 2.26 [95% CI, 1.90-2.69]), TIA (OR, 1.28 [95% CI, 1.09-2.50]), stroke without symptoms (OR, 1.50 [95% CI, 1.19-1.90]), and dialysis (OR, 2.05 [95% CI, 1.24-3.37]).
Predictive Model Application
The predicted risk for stroke, MI, death, or their combination was plotted for the significant individual variables with the highest ORs, as well as for their combination against the patient age (Figure 2). An additive effect of the variables was observed on the risk of all 4 outcomes. For instance, symptomatic patients with angina appear to have double the risk of stroke (4%) in comparison with symptomatic patients without angina (2%; Figure 2A). Also, an 80-year-old patient with previous history of stroke has a much smaller risk of MI (0.5%) than similar patients with additional history of angina and diabetes mellitus (4%). Furthermore, the addition of COPD in the risk factors of symptomatic patients in their 80s with history of MI triples (from 2% to 6%) their risk of death (Figure 2C).
Interestingly, based on this model there was a steep effect of age on the risk of MI and death. Patients with previous history of stroke, angina and diabetes mellitus undergoing CEA in their 60s have a 2% risk of death. This risk appears to double in patients in their 80s (Figure 2B). The risk of death in symptomatic patients with previous history of MI and COPD triples for patients in their 80s in comparison with patients in their 60s (Figure 2C).
Accuracy of the Model
Our model demonstrated moderate discriminative ability. The area under curve or c-statistic of the receiver operating characteristic (Figure I in the online-only Data Supplement) was calculated to be 0.64, 0.70, 0.75, and 0.65 for postoperative stroke, MI, death, and combined risks, respectively. Our validated c-statistics yielded similar values. The cross-validated C-index was 0.63 (SE=0.11), 0.65 (SE=0.016), 0.73 (SE=0.015), and 0.64 (SE=0.008) for postoperative stroke, MI, death, and combined risks, respectively. The bootstrap validated C-index was 0.63 (SE=0.012), 0.66 (SE=0.019), 0.74 (SE=0.016), 0.64 (SE of 0.008) for postoperative stroke, MI, death, and combined risks, respectively. In addition to the area under curve, we performed the Hosmer-Lemeshow test and found that, for 2 of the outcomes (MI and death), our model demonstrated good calibration (P=0.639 and P=0.371 respectively), whereas accuracy of models predicting stroke or combined outcomes yielded P values of 0.011 and 0.034, respectively, indicating that additional factors may be involved in predicting these events that were not included in our study. Our analysis was repeated with only the risk factors that were significant in the univariate analysis and yielded similar results.
The identification of modifiable risk factors associated with poor prognosis and the development of predictive models for outcomes are the cornerstones of defining quality in surgical healthcare delivery. The NSQIP, as a high-quality prospectively collected database, can be very helpful for that purpose.17 Although its results are not strictly representative of the US population at large, the quality of the data from a broad range of academic and private institutions17 performing CEA allows for reliable risk factor identification and modeling. Some groups have previously used the NSQIP database to study the outcomes of CEA.18–20 Their results were either limited to small populations or focused on a particular subgroup of patients, paying attention to their neurological outcomes. They were lacking an analysis of the effect of risk factors on the comprehensive outcomes of CEA, including MI, as is commonplace in most recent RCTs, and did not involve the development of predictive models.
Previous stroke or TIA was recognized in the current analysis as an independent risk factor associated with a higher incidence of stroke, death, or their combination. Interestingly, there is no observed correlation of increased age and the risk of stroke, whereas such a correlation is evident in the risk of MI and death. This is supportive of the notion that CEA should be favored over stenting in older patients. The latter has been shown to have a significantly higher incidence of stroke in the octogenarians, as is evident by the lead-in phase of the Carotid Revascularization Endarterectomy versus Stenting (CREST) trial.21 Our observed 30-day incidence of stroke or death for symptomatic patients (2.3%) is lower but still comparable to what was reported in the CREST trial22 (3.2%) and lower than the corresponding rates in the Stent Protected Angioplasty versus Carotid Endarterectomy (SPACE) trial23 (6.3%), Endarterectomy versus Angioplasty - Symptomatic Severe Carotid Stenosis (EVA-SS)24 (3.9%), and International Carotid Surgery Study (ICSS)25 (3.4%). Similarly the 30-day incidence of stroke in our asymptomatic cohort (1.1%) is comparable to the CREST trial22 (1.3%) and lower than Asymptomatic Carotid Atherosclrerosis Study (ACAS)3 (2.3%) and Asymptomatic Carotid Atherosclrerosis Study (ACST)4 (3.1%). The similar rates of periprocedural stroke in our sample compared with those observed in RCTs suggest that clinical trial-based complication rates can be achieved in nontrial practice.
As expected, there is independent correlation of angina, diabetes mellitus, and coronary angioplasty with the risk of MI. Previous MI does not appear to be independently associated with perioperative MI, probably because of the very strict definition of MI by NSQIP in the perioperative period, which is expected to be different from what was deemed as an MI in the clinical history of the patient. The association of stroke with persistent symptoms with the risk of postoperative MI requires special attention, given its nonconcordance with the lack of such association with TIA and stroke without residual symptoms. This observation can be justified by the fact that symptomatic stroke is probably associated with a larger vascular distribution and potentially a higher general atherosclerotic burden for the patient, contributing to the higher rate of postoperative MI. Our observed incidence of MI was lower than the observed incidence (2.4%) in the CREST study.22 This could have resulted from the stricter definition of postoperative MI in the NSQIP group (defined as a new transmural acute MI occurring during surgery or within 30 days, manifesting by new Q-waves on ECG).
These results are important in the context of the ongoing debate on the appropriateness of CEA or carotid stenting in the treatment of carotid stenosis. Data from the CREST trial22 have supported the use of CEA in patients at higher risk for stroke and carotid stenting in patients at higher risk for an MI. Our proposed model can predict the risk of stroke, MI, death, or the composite outcome for patients in the NSQIP cohort. Although the c-statistic values demonstrated only modest discrimination26 and the models cannot be used for individual patient decision making, they may be of value as a baseline for improvement programs and the development of more accurate predictive models based on other databases or prospective studies. The similarity of rates reported in this analysis, as compared with those observed in RCTs, suggests that the results obtained in the more restricted population enrolled in clinical trials might be generalizable.
Despite providing some projections for all of the carotid stenosis patients, the interpretation of the results in asymptomatic patients will be limited. There is a national trend toward minimizing intervention for asymptomatic carotid stenosis in favor of medical treatment.27 This is driven by a paradigm shift in what constitutes best medical management, from aspirin in the initial RCTs to broad cardiovascular risk factor modification currently. Therefore, the risk predicted in the model cannot be readily compared with the current practice.
The present study has several limitations. The hospitals participating were not a random sample from the United States but were rather motivated to improve the quality of surgical care, although individual surgeons are unaware of the inclusion of their patients in the NSQIP database preoperatively. Hospitals participating in NSQIP are expected to be larger and more likely to have an academic affiliation than the average US hospital. However, the hospitals included were still diverse with respect to size, region, and academic status, supporting the generalizability of our findings. In addition, the current analysis contains the largest cohort of NSQIP patients investigated to date. The presented model demonstrates relatively poor discrimination for stroke but improved and acceptable discrimination for MI and death. In fact, the NSQIP database contains more general risk factors that are closely related to death or MI as an outcome and is lacking procedure specific risk factors that would provide better discrimination for specific postoperative complications such as stroke.
Several issues also arise with the definition of symptomatic patients that was used. The NSQIP database does not provide data on the timing of the stroke or its etiology. The interval between symptom onset and intervention, particularly for those with moderate symptomatic stenosis, is critical for determining the balance of risks and benefits.28 Therefore, categorizing patients with a previous history of stroke as symptomatic in this setting overestimates the prevalence of symptoms. Although this is a well-known pitfall of NSQIP, we categorized patients as symptomatic in agreement with all of the published literature in this database. However, the proportion of asymptomatic patients in this study is similar to that of regional registries, making the designation of symptomatic status likely accurate. On the other hand, the diagnosis of postoperative stroke is not necessarily based on postoperative examination by an independent neurologist certified on the National Institutes of Health Stroke Scale to identify the severity of the stoke, and there is no mention of Rankin scores. Therefore, the diagnosis might vary among providers. However, this is most likely representative of a real world experience, where a stroke might be diagnosed by the provider performing the procedure (ie, a neurosurgeon). In addition, there is no mention of sidedness, contralateral carotid occlusion, irregular or ulcerated plaques in the angiogram, or hemispheric versus retinal TIAs, which were all found to be risk factors in North American Symptomatic Carotid Endarterectomy Trial (NASCET).5 There is also no record of the use of statins or antiplatelet therapy, the history of atrial fibrillation, the timing of the intervention, or the technical details of the procedure. Finally, to create our multivariate model, we considered all of the preoperative patient-level risk factors available in the NSQIP database that are associated with vascular disease. Some risk factors that were thought to be unrelated or overlapping with others were excluded from the model. This introduces a slight selection bias.
The NSQIP database is a prospective database, appropriate for benchmarking in the quality of national healthcare delivery. Based on the results of the multivariate analysis, a risk factor-based predictive model of the postoperative 30-day risk of stroke, MI, death, or their combination was devised. Although the generalization of these predictions should be made with caution, the model can be used as a baseline for the improvement and development of more accurate predictive models based on other databases or prospective studies, but the level of discrimination is insufficient for use in individual patient decision making.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.111.674358/-/DC1.
- Received August 17, 2012.
- Revision received January 9, 2013.
- Accepted January 18, 2013.
- © 2013 American Heart Association, Inc.
- Halliday A,
- Mansfield A,
- Marro J,
- Peto C,
- Peto R,
- Potter J,
- et al
- Wong JH,
- Findlay JM,
- Suarez-Almazor ME
- Bond R,
- Rerkasem K,
- Rothwell PM
- Rothwell PM,
- Slattery J,
- Warlow CP
- Goldstein LB,
- McCrory DC,
- Landsman PB,
- Samsa GP,
- Ancukiewicz M,
- Oddone EZ,
- et al
- Davenport DL,
- Holsapple CW,
- Conigliaro J
- Woo K,
- Garg J,
- Hye RJ,
- Dilley RB
- Hopkins LN,
- Roubin GS,
- Chakhtoura EY,
- Gray WA,
- Ferguson RD,
- Katzen BT,
- et al
- Ringleb PA,
- Allenberg J,
- Brückmann H,
- Eckstein HH,
- Fraedrich G,
- Hartmann M,
- et al
- International Carotid Stenting Study investigators,
- Ederle J,
- Dobson J,
- Featherstone RL,
- Bonati LH,
- van der Worp HB,
- et al