Published online before print April 29, 2004, doi: 10.1161/01.STR.0000128706.41021.01
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Stroke. 2004;35:1399.)
© 2004 American Heart Association, Inc.
Original Contributions |
Impact of Surgical Clipping on Survival in Unruptured and Ruptured Cerebral Aneurysms
A Population-Based Study
From the Departments of Neurosurgery (G.W.B., D.W.N., J.E.), Radiology (G.W.B., J.E.), Surgery (L.S., D.R.F.), and Health Services (D.R.F.), University of Washington, Seattle, Wash.
Correspondence to Dr David R. Flum, Assistant Professor, University of Washington, Department of Surgery, BB 431, 1959 NE Pacific St, Box 356410, Seattle, WA 98195-6410. E-mail daveflum{at}u.washington.edu
 |
Abstract |
|---|
 Top Abstract IntroductionPatients and MethodsResultsDiscussionReferences |
|---|
Background and Purpose— The management of aneurysms is controversial because little is known about the impact of clipping on long-term outcome. This study was designed to evaluate long-term survival of patients with aneurysms undergoing clipping in a statewide population.
Methods— We used a retrospective design using an administrative database to identify patients hospitalized with aneurysms (1987 to 2001). Time-to-event analysis was used to determine the risk of death from all causes and from neurological causes.
Results— 4619 patients (mean age 54.7±15.3, 66.3% female) were hospitalized with cerebral aneurysms. Survival among patients with ruptures was significantly lower compared with patients with unruptured aneurysm (P<0.001) with adjusted hazard ratio (HR) of death after clipping 40% higher (HR: 1.4; 95% CI: 1.2, 1.7) in patients with rupture compared with those that were unruptured. Survival estimates for unruptured patients who underwent clipping were significantly higher than among those unruptured patients who did not undergo clipping (P<0.001), with adjusted HR of death 30% higher in patients with unruptured aneurysm that were not clipped compared with unruptured patients who were clipped (HR: 1.3; 95% CI: 1.1, 1.6). Patients with unruptured aneurysm who underwent clipping and survived beyond the 30-day postoperative period were less likely to die from neurologically related causes (5.6 versus 2.3%, P<0.001). Patients with ruptures and aneurysms who underwent clipping have a higher rate of death compared with the general population in the long-term.
Conclusions— Short-term and long-term mortality after clipping of cerebral aneurysms is higher than previously reported. Patients with unruptured aneurysms who undergo clipping have improved survival compared with those who do not undergo clipping. This study supports the use of early intervention in the management of patients with unruptured aneurysms.
Key Words: intracranial pressure • aneurysm • surgery • epidemiology • outcome
|
Introduction |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsDiscussionReferences |
|---|
More than 700 000 cerebrovascular accidents (CVAs) occur yearly, making it the third leading cause of death in the United States.1,2 As many as 15% of all CVAs are secondary to ruptured aneurysms,3,4 and CVAs related to aneurysms are associated with 30-day mortality rates of between 45% and 80%.5,6 Half of all survivors of ruptured aneurysms sustain irreversible brain damage, 5–7 and although most stop bleeding spontaneously, 50% rebleed within 6 months.3,8 Rebleeding of an aneurysm is associated with a 50% to 85% risk of death,3,8–11 and exclusion of the aneurysm from the circulatory system by microsurgical clipping or endovascular coiling has been recommended to reduce this risk. Surgical clipping, the better-studied of the interventions for aneurysms, has been demonstrated to completely obliterate the aneurysm in >90% of patients,12,13 but its impact on long-term survival in patients with rupture has not been well addressed. The availability of noninvasive brain imaging has resulted in an increase in the diagnosis of asymptomatic unruptured cerebral aneurysms. The management of unruptured aneurysms remains controversial because of incomplete and conflicting data about the natural history of these lesions and the risk associated with clipping.3,14–16 For example, although 2 recent, prospective evaluations of the natural history of unruptured aneurysms and the risks of treatment demonstrated a 1.8% to 2.3% 30-day mortality rate and a 2.7% to 3.8% 1-year mortality rate after surgical clipping,17,18 a recent meta-analysis suggested that the rate of early mortality may be as high as 7%. As with ruptured aneurysms, little is known about the impact of clipping of unruptured aneurysms on long-term outcome and accurate determinations of short-term and long-term outcome are critical in counseling patients and developing clinical algorithms.
To better counsel patients and assess the relative benefits of early intervention, this study was designed to evaluate long-term survival of patients with ruptured and unruptured cerebral aneurysms undergoing clipping in a statewide population and to evaluate outcome differences among patients with unruptured aneurysms who did and did not undergo aneurysm clipping.
|
Patients and Methods |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsDiscussionReferences |
|---|
Study Design
A retrospective cohort design was used, using a statewide administrative discharge database to determine the impact of surgical clipping on patient survival in the state of Washington from 1987 to 2001.
Setting
Data were obtained from the Washington State Comprehensive Hospital Abstract Reporting System (CHARS) database. This data set is derived from all public and private hospitals in Washington State (Veterans Affairs and US military hospitals excluded). It contains demographic variables, admission and discharge administrative details, payer status, International Classification of Diseases, Ninth Revision (ICD-9) procedure and diagnosis codes, and hospital identifiers. This study was granted an exception by agreement of the University of Washington Human Subject Review Committee and the Washington State Department of Health. The data set includes only anonymous data and is considered within the public domain.
Subjects
All CHARS reports from January 1, 1987, through December 31, 2001, were searched for ICD-9 diagnostic and procedure codes pertaining to ruptured (430) and unruptured cerebral aneurysms (437.3).
Variable Definitions and Analysis
Patients were divided into 3 groups: group 1, those with ruptured aneurysms who underwent surgical clipping (39.51); group 2, those with unruptured aneurysms who underwent surgical clipping (39.51); and group 3, those with unruptured aneurysms who were not treated with surgical clipping.
A modified Charlson comorbidity index19 (0 to 3, with 3 indicating greatest comorbidity) was calculated for each patient based on ICD9 diagnostic codes. Four age groups were defined: 18 to 35 years, 36 to 55 years, 56 to 75 years, and older than 76 years. "All-cause" death was considered an event. The rate of neurologically related causes of death was calculated for unruptured patients undergoing differential management. This variable was defined as patients having the following ICD-9 codes as their cause of death: 430, 433.1, 434.1, 434.9, 435, 436, 437.0, 437.3, 437.9, 438, 290.1, 291.8, 331.0, 331.4, 332.0, 340, and 348.8.
Analysis
Descriptive and comparative statistics were applied using STATA Version 7 (STATACorp). Categorical variables were compared using Pearson 
2 statistic, and continuous variables were evaluated using ANOVA. Kaplan-Meier survival estimates were used to evaluate the differential effect of ruptured compared with unruptured aneurysms on survival after clipping and the differential effect of clipping and nonclipping on patients with unruptured aneurysm. All-cause death was the event of interest. December 31, 2001, was the censoring date for those who survived during follow-up. The median time at risk was 1.9 years (mean 3.7 years, range 0 to 14 years with 25% followed-up for >6.4 years). Log rank and Wilcoxon testing were used to compare unadjusted survival estimates to determine the equality of survival curves. Cox proportional hazards were used to simultaneously control for multiple covariates while determining the hazard ratio (HR) of death as it related to ruptured versus unruptured clipped aneurysms and death as it related to clipping or no clipping among patients with unruptured aneurysms. The proportional hazards assumption was confirmed by inspecting Schoenfeld residuals. Proportional hazard models were constructed in 2 steps. Model 1 analyses (unadjusted) included only the primary predictor of interest whereas model 2 analyses included the primary predictor and demographic variables such as older age groups, sex, Charlson comorbidity index, and Medicaid status.
For descriptive benefit, expected survival estimates in the population at large were derived from the vital statistics database.20
|
Results |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsDiscussionReferences |
|---|
4619 patients (mean age 54.7±15.3, 66.3% female) were hospitalized with cerebral aneurysms during the study period 1987 to 2001. Patients were divided into 3 groups: those with ruptured aneurysms who underwent surgical clipping (group 1, n=2465), those with unruptured aneurysms who underwent surgical clipping (group 2, n=1062), and those with unruptured aneurysms who were not treated with surgical clipping (group 3, n=1092). Demographic features (Table 1) appeared unique in group 3 (unruptured, unclipped patients) compared with the 2 other groups (clipped ruptured or clipped unruptured patients). Patients in group 3 were significantly older, more likely to be male, had a higher average comorbidity index, were less likely to have commercial insurance. Group 3 patients also had shorter lengths of stay and significantly lower hospital charges. Conversely, patients with ruptured aneurysms undergoing clipping had the longest lengths of hospitalizations and associated hospital charges.
|
The 30-day mortality rates were highest among clipped patients with ruptured aneurysms compared with those with unruptured aneurysms (13.4% versus 5.5%, respectively), and this differential was noted at all follow-up points (17.9% versus 8.5%, 22.4% versus 13.4%, 29% versus 24%, at 1, 5, and 10 years, respectively). Survival among ruptured patients was significantly lower compared with patients with unruptured aneurysm (P<0.001), but after the first-year postclipping, differences in the survival rate appeared to be diminished (FigureA). The adjusted HR of death after clipping was 40% higher (HR 1.4; 95% CI: 1.2, 1.7) in patients with ruptures compared with those unruptured (Table 2A).
|
|
When comparing patients with unruptured aneurysms, those who underwent clipping had lower rates of 30-day mortality (5.5% versus 7.6%, respectively) and this differential was noted at all follow-up points (8.5 versus 16.8%, 13.4% versus 30%, 24% versus 44.5%, at 1, 5, and 10 years, respectively). Survival estimates for unruptured patients who were clipped were significantly higher than among those unruptured patients who were not clipped (P<0.001), but the rate of change of these estimates appeared to diminish over time (FigureB). Patients with unruptured aneurysm who underwent clipping (and survived beyond the 30-day postoperative period) were significantly less likely to die from neurologically related causes (including cerebrovascular accidents and subarachnoid hemorrhage). The rate of neurologically related death was 5.6% versus 2.3% in unclipped versus clipped patients with unruptured aneurysm, respectively (P<0.001) The adjusted HR of death was 30% higher in patients with unruptured aneurysm that were not clipped (HR 1.3; 95% CI: 1.1, 1.6) compared with unruptured patients who were clipped (Table 2B).
|
When incremental changes in the percentage alive after intervention were compared with a population estimate (not sex-standardized), patients with both ruptured and nonruptured aneurysms undergoing clipping appeared to have a higher incremental rate of death between discrete follow-up points (including 10-year follow-up) than the population at large (Table 3). In general, the incremental rate of death per year was between 30% and 240% higher in patients undergoing clipping than in the population at large, and in only 1 year was the incremental ratio 1.
|
|
Discussion |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsDiscussionReferences |
|---|
In this study, we found that patients undergoing clipping of ruptured aneurysms had lower short-term and long-term survival compared with clipped patients with unruptured aneurysms. Patients with unruptured aneurysms had a significantly higher rate of mortality when not undergoing clipping compared with unruptured patients who were clipped. Patients with any type of aneurysm undergoing a clipping procedure appeared to have a higher year-by-year incremental rate of mortality compared with the population at large. This effect appeared to extend to as long as 10 years after clipping.
Population-based research is an important adjunct to outcomes assessment because by studying large communities of patients, the population frequency of adverse outcome can be determined. These rates often differ from the findings of smaller case series because they reflect a more "real-world" appraisal of patient selection, risk, and variation in practice effectiveness. Only 1 population-based appraisal of aneurysm intervention has been reported21 among the several reports of short-term22 or long-term impact of aneurysmal therapy.21,23 In a study evaluating a large population in Finland, Ronkinen et al21 found that patients treated after an aneurysmal rupture had a 1-year mortality rate of 12.6%. Our study found this rate to be 17.9%, nearly 50% higher, and it is unclear what may have accounted for these differences. This difference may reflect the fact that the operations in the Finnish study were performed in a referral center, with large volume of operations performed by expert surgeons. Hospital procedural volume has been associated with better outcomes.24 Although it is commonly considered that after clipping of ruptured aneurysms patients are safe from possible aneurysm rebleeding and therefore have a normal life expectancy, several long-term reports have suggested sustained decrements in survival.23,25 Survival beyond the immediate postoperative state has several components, including the long-term effects of the initial hemorrhage and treatment and rehabilitation. The Finish study found that most deaths after long-term follow-up were attributable to cardiovascular disease. The authors postulated that cerebral aneurysms may be a marker for more generalized vascular disease. Our study could not adequately assess the risk of cardiovascular disease because of constraints of administrative data, but we did identify a 30% to 220% higher rate of yearly incremental mortality in patients undergoing clipping as long as 10 years after the intervention.
In ruptured aneurysms, the decision to intervene may be straightforward; however, for unruptured aneurysm, the question is often whether to perform a clipping procedure. In 2 recent meta-analyses, the 30-day risk of dying after clipping was found to be 1% to 7% with combined morbidity of 4% to 15%.26,27 Two recent, prospective evaluations of the natural history of unruptured aneurysms and the risks of treatment demonstrated a 1.8% to 2.3% 30-day mortality and 2.75 to 3.8% 1-year mortality after surgical clipping.17,18 Our study demonstrated a 30-day mortality rate of 5.5% and 1-year mortality of 8.5%, both double the estimates of these prospective studies and in the upper range reported in the meta-analyses. The higher rates that we found may reflect the results of lower procedural volume hospitals and should be considered a "real-world" appraisal. Another reason for the higher observed mortality may be because of reporting bias; series with inferior outcomes may not be reported. Also, selection bias that may be present in case series; patients with fewer comorbidities may be the ones who are operated on and thus have better outcomes. The question of whether unruptured aneurysms do better with or without clipping is controversial because the natural history of these lesions is unclear, as are the risks of clipping them. So far, all natural history studies have been performed only on selected patients, a fact that may have influenced the results. Accumulating evidence points to an influence of aneurysm size on the risk of rupture in patients with unruptured aneurysms, with larger lesions more likely to hemorrhage.3 Importantly, in none of these previous studies was there an adequate comparator group to determine relative survival benefits of the clipping procedure for patients with unruptured aneurysm. In fact, when compared with unruptured patients undergoing clipping we found that those not undergoing clipping were 30% more likely to die than were clipped patients. Although patients not selected for clipping may be different in important ways not assessed in this study, so-called confounding by indication, we believe these data support the growing body of literature that advocates earlier intervention in patients with unruptured aneurysms.
This study has several limitations. In assessing outcome for patients with unruptured aneurysms that were or were not clipped, confounding by indication (for clipping or no clipping) is relevant. The question of confounding by indication is difficult to assess in this population and may in fact act as a conservative bias. Patients who did not undergo clipping were older, had more comorbidities, were more likely male, and were more likely to have Medicare/Medicaid as their payer; all of these were associated with worse outcomes. Patients may not have been directed to intervention because of perceived risk and that might explain the increased hazard ratio in unclipped, unruptured patients. Furthermore, Medicaid as a payer, a proxy of socioeconomic status,28,29 was associated with a poor outcome, and this may be related to delayed diagnosis, poor access to health care, and other components that may be associated with decreased survival. Alternatively, patients with unruptured aneurysm are often treated if the risk of rupture is high compared with patients with a "low risk" of rupture.3 These "more likely to rupture" patients may be at higher risk for interventions than are lower-risk patients. We attempted to adjust for both possibilities using proportional hazards modeling and found that even after controlling for age and Charlson index, the HR was 30% higher in unclipped patients. Another limitation is that quality of life and subtle morbidity data could not be obtained. Surgical clipping is associated with a morbidity rate up to 15%, 17,18,26,27 which was not considered in this outcome analysis. Furthermore, in the group that had a ruptured aneurysm, we were limited in available descriptors of the clipping procedure. For example, the data set does not detail if the clipped aneurysm bled or if a separate unruptured aneurysm bled. Lastly, important clinical variables were also unavailable for this analysis. For example, the risk of aneurysmal rupture is directly related to the diameter.18 The benefit of elective clipping may be more pronounced in the large-diameter groups, because the increased risk of rupture offsets the risks of surgery. Such analysis cannot be performed using the administrative database, because the information about the diameter of the aneurysm is not coded.
In conclusion, this largest-ever evaluation of statewide outcomes after clipping of cerebral aneurysm demonstrated higher than previously reported rates of short-term and long-term mortality after clipping. This study also demonstrates improved survival among patients with unruptured aneurysms who undergo clipping compared with those who do not undergo clipping, and a significant decrease in neurologically related causes of death in the treated group. Lastly, this study suggests persistent decrements in survival over time after aneurysmal clipping. This work supports the theory that patients with aneurysm remain at higher risk for all-cause mortality, but the source of these decrements in survival is not completely understood. Counseling patients with aneurysms remains a challenge, but this work supports the use of early detection and intervention in their management.
Received January 8, 2004; revision received March 1, 2004; accepted March 4, 2004.
|
References |
|---|
|
TopAbstractIntroductionPatients and MethodsResultsDiscussionReferences |
|---|
1. Warlow CP. Epidemiology of stroke. Lancet. 1998; 352 (suppl 3): SIII1–SIII4.[Medline] [Order article via Infotrieve]
2. Broderick JP. Practical considerations in the early treatment of ischemic stroke. Am Fam Physician. 1998; 57: 73–80.[Medline] [Order article via Infotrieve]
3. Bederson JB, Awad IA, Wiebers DO, Piepgras D, Haley EC Jr, Brott T, Hademenos G, Chyatte D, Rosenwasser R, Caroselli C. Recommendations for the management of patients with unruptured intracranial aneurysms: a statement for healthcare professionals from the Stroke Council of the American Heart Association. Stroke. 2000; 31: 2742–2750.
4. Bederson JB, Awad IA, Wiebers DO, Piepgras D, Haley EC Jr, Brott T, Hademenos G, Chyatte D, Rosenwasser R, Caroselli C. Recommendations for the management of patients with unruptured intracranial aneurysms: a statement for healthcare professionals from the Stroke Council of the American Heart Association. Circulation. 2000; 102: 2300–2308.
5. Broderick JP, Brott TG, Duldner JE, Tomsick T, Leach A. Initial and recurrent bleeding are the major causes of death following subarachnoid hemorrhage. Stroke. 1994; 25: 1342–1347.[Abstract]
6. Hyland H. Prognosis in spontaneous subarachnoid hemorrhage. Arch Neurol Psy. 1950; 63: 61–78.
7. Kassell NF, Torner JC. Size of intracranial aneurysms. Neurosurgery. 1983; 12: 291–297.[Medline] [Order article via Infotrieve]
8. Winn HR, Richardson AE, Jane JA. The long-term prognosis in untreated cerebral aneurysms: I. The incidence of late hemorrhage in cerebral aneurysm: a 10-year evaluation of 364 patients. Ann Neurol. 1977; 1: 358–370.[CrossRef][Medline] [Order article via Infotrieve]
9. Nishioka H, Torner JC, Graf CJ, Kassell NF, Sahs AL, Goettler LC. Cooperative study of intracranial aneurysms and subarachnoid hemorrhage: a long-term prognostic study. III. Subarachnoid hemorrhage of undetermined etiology. Arch Neurol. 1984; 41: 1147–1151.
10. Nishioka H, Torner JC, Graf CJ, Kassell NF, Sahs AL, Goettler LC. Cooperative study of intracranial aneurysms and subarachnoid hemorrhage: a long-term prognostic study. II. Ruptured intracranial aneurysms managed conservatively. Arch Neurol. 1984; 41: 1142–1146.
11. Le Roux PD, Elliot JP, Newell DW, Grady MS, Winn HR. The incidence of surgical complications is similar in good and poor grade patients undergoing repair of ruptured anterior circulation aneurysms: a retrospective review of 355 patients. Neurosurgery. 1996; 38: 887–893;discussion 893–895.[CrossRef]
12. David CA, Vishteh AG, Spetzler RF, Lemole M, Lawton MT, Partovi S. Late angiographic follow-up review of surgically treated aneurysms. J Neurosurg. 1999; 91: 396–401.[Medline] [Order article via Infotrieve]
13. Le Roux PD, Elliott JP, Eskridge JM, Cohen W, Winn HR. Risks and benefits of diagnostic angiography after aneurysm surgery: a retrospective analysis of 597 studies. Neurosurgery. 1998; 42: 1245–1254.
14. Winn HR, Jane JA Sr, Taylor J, Kaiser D, Britz GW. Prevalence of asymptomatic incidental aneurysms: review of 4568 arteriograms. J Neurosurg. 2002; 96: 43–49.[Medline] [Order article via Infotrieve]
15. Aoki N, Beck JR, Kitahara T, Ohbu S, Soma K, Ohwada T, Cone RW, Fukui T. Reanalysis of unruptured intracranial aneurysm management: effect of a new international study on the threshold probabilities. Med Decis Making. 2001; 21: 87–96.
16. Brennan JW, Schwartz ML. Unruptured intracranial aneurysms: appraisal of the literature and suggested recommendations for surgery, using evidence-based medicine criteria. Neurosurgery. 2000; 47: 1352–1371.[CrossRef]
17. Wiebers DO, Whisnant JP, Huston J 3rd, Meissner I, Brown RD Jr, Piepgras DG, Forbes GS, Thielen K, Nichols D, O’Fallon WM, Peacock J, Jaeger L, Kassell NF, Kongable-Beckman GL, Torner JC. Unruptured intracranial aneurysms: natural history, clinical outcome, and risks of surgical and endovascular treatment. Lancet. 2003; 362: 103–110.[CrossRef][Medline] [Order article via Infotrieve]
18. Investigators ISoUIA. Unruptured intracranial aneurysms - risk of rupture and risks of surgical intervention. N Engl J Med. 1998; 339: 1725–1733.
19. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992; 45: 613–619.[CrossRef][Medline] [Order article via Infotrieve]
20. Statistics VaH. Life table for the total population. United States, 1996. National Center for Health Statistics. 1996; 2: 12.
21. Ronkainen A, Niskanen M, Rinne J, Koivisto T, Hernesniemi J, Vapalahti M. Evidence for excess long-term mortality after treated subarachnoid hemorrhage. Stroke. 2001; 32: 2850–2853.
22. Osawa M, Hongo K, Tanaka Y, Nakamura Y, Kitazawa K, Kobayashi S. Results of direct surgery for aneurysmal subarachnoid haemorrhage: outcome of 2055 patients who underwent direct aneurysm surgery and profile of ruptured intracranial aneurysms. Acta Neurochir (Wien). 2001; 143: 655–663;discussion 663–644.
23. Kaste M, Troupp H. Subarachnoid haemorrhage: long-term follow-up results of late surgical versus conservative treatment. BMJ. 1978; 1: 1310–1311.
24. Berman MF, Solomon RA, Mayer SA, Johnston SC, Yung PP. Impact of hospital-related factors on outcome after treatment of cerebral aneurysms. Stroke. 2003; 34: 2200–2207.
25. Olafsson E, Hauser WA, Gudmundsson G. A population-based study of prognosis of ruptured cerebral aneurysm: mortality and recurrence of subarachnoid hemorrhage. Neurology. 1997; 48: 1191–1195.
26. King JT Jr, Berlin JA, Flamm ES. Morbidity and mortality from elective surgery for asymptomatic, unruptured, intracranial aneurysms: a meta-analysis. J Neurosurg. 1994; 81: 837–842.[Medline] [Order article via Infotrieve]
27. Raaymakers TW, Rinkel GJ, Limburg M, Algra A. Mortality and morbidity of surgery for unruptured intracranial aneurysms: a meta-analysis. Stroke. 1998; 29: 1531–1538.
28. Bach PB, Guadagnoli E, Schrag D, Schussler N, Warren JL. Patient demographic and socioeconomic characteristics in the SEER-Medicare database applications and limitations. Med Care. 2002; 40: IV-19–25.
29. Braveman P, Cubbin C, Marchi K, Egerter S, Chavez G. Measuring socioeconomic status/position in studies of racial/ethnic disparities: maternal and infant health. Public Health Rep. 2001; 116: 449–463.[CrossRef][Medline] [Order article via Infotrieve]
This article has been cited by other articles:
 |
|
 |
|
  J. Raymond, J. Silvaggio, F. Guilbert, A. Weill, and D. Roy Impact of Surgical Treatment of Unruptured Aneurysms Stroke, October 1, 2005; 36(10): 2069 - 2070. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||