Using Centrally Held Data to Validate Carotid Surgery Outcome Data
Background and Purpose—Outcome audit data for peer group comparison must be transparent, objective, and independently reproducible. Personal data sets are difficult to maintain and often lack complete follow-up. Local coding difficulties make initial retrieval of centrally held data unreliable. However starting with a complete list of interventions, reliable identification of patients who have experienced an adverse postoperative event may be possible using record linkages.
Methods—A surgical database, augmented by a hand-search of all theater registries and personal logbooks, identified 378 carotid endarterectomies performed for stroke prevention in symptomatic patients, in a single hospital between 2002 and 2009. A list of the names, unique patient identifiers, and operation dates was sent to the Information Services Division of National Health Service Scotland. Data were requested pertaining to all deaths and potential diagnoses of stroke after surgery. Every identified case was scrutinized.
Results—There were 30 (8%) readmissions or transfers of care identified within 30 days of surgery. From this, 12 strokes were identified with another 2 strokes, occurring without readmission, diagnosed in the outpatient clinic. Only 6 of the postoperative strokes were identified during the index admission. There were 2 early deaths resulting in a combined stroke and death rate of 4.2% (95% confidence intervals, 2.4%–6.9%).
Conclusions—These outcome data are similar to the outcomes of the major carotid surgery trials. Record-linked data retrieval seems to be an appropriate starting point for outcome-based audit. This has the potential to generate robust, transparent data for comparison between individuals and centers for a specific procedure.
- carotid endarterectomy
- symptomatic carotid stenosis
- vascular surgery
- outcomes research
Clinical audit serves a variety of purposes. Accurate identification of complication rates is desirable for the provision of informed consent and to drive improvements in the quality of patient care. Outcome data may also be used to scrutinize units or individuals, measuring performance within peer groups. It would be wrong simply to accept complications as an inevitability of surgery, and a considered approach recognizes a need to identify complications correctly and then to determine whether an appropriate response was instituted and whether changes in care pathways or practices are necessary to reduce procedural risk. An unhealthy attitude to the diagnosis of complications not only results in a reluctance to search for, diagnose, and document an adverse event, but may also lead to inappropriate and inadequate remedial action, further disadvantaging the patient.1–3
The indications for surgical intervention for symptomatic carotid artery disease have been clearly defined by several major randomized controlled trials (RCTs) and meta-analyses.4 Derived from these collated trial data are the range of outcomes providing acceptable complication rates for nontrial practice. Surgical outcomes may have improved with changes in perioperative management, particularly the use of newer antiplatelet agents and high-dose statin therapy, but robust contemporary data suggest that complication rates remain largely similar to older studies.5 What is also clear is that depending on the method of audit there exist a wide range of documented outcomes within the literature.6
Although the wealth of quality data pertaining to carotid endarterectomy offers a benchmark for outcome audit, a methodology that reliably identifies major adverse outcomes, without the context of a major trial, is lacking. Postoperative death represents simple categorical data, but there is potential for ambiguity with respect to stroke. The diagnosis, nature, and severity of procedure-related stroke are not uniformly defined,6,7 an issue that to some extent has been compounded by variation in reporting standards between the various trial collaborative groups.8–10 For example, the North American Symptomatic Carotid Endarterectomy Trial (NASCET) Collaborators,9 used Committee on Classification of Cerebrovascular Disease of the National Institute of Neurological Disorders and Stroke III, whereas the European Carotid Surgery Trialists’ (ECST) Collaborative Group10 collaborators adopted the Modified Rankin criteria. Furthermore, there were clear differences in the timing of assessment of stroke severity (3 months by NASCET9 and 6 months by ECST).10
A well-designed clinical research trial demands either independent review of all cases or outcome measures with hard end points beyond dispute, and ideally both. However, in the absence of adequate resource to recreate the methods of trial data capture, outcome audit within the constraints of routine practice may be compromised in terms of quality, completeness, and, therefore, reliability. In current healthcare management, large volumes of data are routinely collected at regional or corporate levels and are collated in centrally held databases. Within the United Kingdom, these are controlled by the National Health Service, in other healthcare systems various management organizations will collect these data. In Scotland, collated data include diagnosis (International Classification of Disease-Tenth Revision, ICD-10) and treatment codes (Office of Population Census and Surveys classification of surgical operations and procedures, OPSC-4), as well as the date and location of all National Health Service admissions and the date and mode of discharge. The use of routinely collected medical record data in identifying serious adverse events in comparison with traditional clinical trial approaches has been evaluated previously.11 Interestingly, the finding was that both approaches had advantages and disadvantages, with a significant degree of concordance in the events identified. In addition, Ford et al12 showed that whether the West of Scotland Coronary Prevention Study (WOSCOPS) trial had been run using only record-linkage outcomes, all the key findings of the study would have been qualitatively identical.
For a variety of reasons, personal carotid surgery audit seems particularly vulnerable to incomplete data sets. Patients experiencing a major neurological complication are rapidly transferred to a stroke care team for treatment and rehabilitation. This results in a potential failure to generate a surgical discharge summary and also results in discharge coding from a different team. The majority of carotid endarterectomy patients are discharged soon after surgery and a significant proportion of early postoperative events will occur in the community. This may result in a readmission to a stroke team rather than the surgical team perhaps in a different hospital, or even heath board. As such, the complication may never be communicated to the surgical team. Similarly, the outpatient visit cannot be relied on to complete data collection, as there are a variety of reasons for nonattendance, including a disabling complication. Another issue that cannot be ignored in the follow-up process is a lack of resource to ensure complete outcome data.
Various National Health Service statistics are compiled throughout the United Kingdom, using slightly different methods of data collection dependent on location. In Scotland, the data are centrally collated by the Information Services Division (ISD) as the Scottish Morbidity Record. The codes of 1 or more diagnoses are collated and assigned to each hospital episode, similar to Hospital Episode Statistics data in England and Wales. Some criticism could be made of this process of data capture as coding errors do occur, most frequently in cases of greater complexity and those associated with inter- or intrahospital transfers. However within the Scottish health service, the generation of a single and unique Community Health Index (CHI) number as an identifier for each patient allows tracking of cases, with the ability to retrieve data on (re)admission or death after any specified date (eg, the date of procedure).
This study aims to examine the process of retrieving centrally held data with event identification supported by clinical review. We sought to identify significant adverse events, particularly stroke and death, and make comparison with trial data. A secondary aim at the outset was to document the long-term survival of operated patients, as a surrogate marker of patient selection.
A single-unit carotid surgery database had been created in 2004. Initially, this was populated with data gathered retrospectively, but thereafter prospectively filled, to cover 2002 to 2009. Cases were gathered from, and cross-checked against, multiple sources, including personal logbooks, theater records, secretarial diaries, and a combined physician-referral/surgical audit datasheet completed with each discharge summary. Initial follow-up data were obtained by examining inpatient and outpatient records from the surgical and stroke physician clinics.
Having sought to identify every operative case, the list of patients’ names, CHI number, and operation date was sent to the ISD of National Health Service Scotland. ISD provides a double check on the veracity of the CHI by checking against name and date of birth. This process may identify a small number of cases where the CHI requires to be verified manually and corrected if necessary.
ISD have a Privacy Advisory Committee process within their data governance group to oversee use and storage of patient identifiable information. Approval was granted from this group before release of the requested data.
Data were requested on all deaths after carotid endarterectomy and any readmissions that may potentially have been because of a postoperative neurological event. To this end, readmission data were requested to cover neurological conditions, collapse, falls, operative complications, and geriatric falls (colloquially described as "off legs"). Every admission identified by ISD occurring within 90 days of surgery was examined in detail using hospital records and, if clarification was required, by contacting general practitioners. A decision was then made on a case-by-case basis about the likelihood of a stroke having occurred. Helpfully, transfer to another ward, for instance a stroke unit, was recorded as a readmission. This also allowed validation of the data already held with respect to immediate complications.
Retrospectively assigning a modified Rankin score13 proved difficult, except for a minority of cases in which documentation by a stroke physician was identified within the records. Furthermore, even these scores may be subject to challenge.7 Given this, and the range of reporting standards in the literature, a decision was made to count all clear and clinically significant strokes. These were defined as evidence of a persisting neurological deficit with a significant impact on the patient’s life, approximating to a modified Rankin score of ≥2. This was established by reviewing individual case records seeking evidence of a new neurological deficit in keeping with a diagnosis of acute stroke, indicated by a statement on a discharge letter and imaging available on the National Archive. A subsequent consensus was reached between the investigators, on the basis of the evidence identified. Patients were not requested to attend for clinical evaluation. In cases of ambiguity, direct contact with the relevant physicians was planned, but subsequently did not prove necessary.
Survival data are tabulated as counts and percentages and survival event-free curves are created using the method of Kaplan and Meier.
Four hundred two procedures were performed on 393 patients. The median age of the patients was 70, range 40 to 89 years, with 62% male. The indications for surgery and the referral sources are shown (Table 1). The times from presenting symptoms to surgery are indicated (Figure 1).
There were 378 procedures performed for symptomatic disease. The further analyses apply only to these patients. Within the first 30 days of surgery, 14 patients (3.7%; 95% confidence interval, 2%–6.2%) had a stroke. Nine of these were ipsilateral ischemic strokes, 1 was an intracranial bleed and the nature of 3 was indeterminable from correspondence and recorded investigations. One stroke proved fatal. This ischemic stroke occurred during the night after the procedure and affected the contralateral territory. The patient died after 36 days.
Only 2 deaths occurred within the first 30 days; 1 was in a high-risk patient who died of cardiac failure 3 weeks after surgery, and the other death occurred suddenly at home 5 days after surgery. The surgical indication for the former patient was multiple transient ischemic attacks on maximum medical therapy. The postmortem examination of the latter patient indicated that death was likely to be because of ischemic heart disease. The combined figure for significant stroke and death was, therefore, 16 patients in 378 procedures or 4.2% (95% confidence interval, 2.4%–6.9%).
There were 30 readmissions or transfers of care (8%) identified by ISD within 30 days of surgery. Eight patients had strokes diagnosed and treated after discharge from inpatient surgical care. The other 6 strokes were identified during the index surgical admission.
Two of the patients in our series experienced a postoperative stroke but were not seen at, or readmitted to, any hospital after surgery. The diagnosis was made in the surgical outpatient clinic, the patients having been managed by their general practitioners, in the community.
All the inpatient events and readmission strokes that we had been aware of were also identified from the returned ISD data, together with 4 others of which the surgical team was not aware. The 2 patients who had not been readmitted to the hospital after a stroke were not recorded on the ISD data.
A survival plot was drawn (Figure 2) and a table was generated to estimate the 5-year survival rate (81.1%) and the rate of death of the patients calculated as a monthly rate (0.32% death rate per month). The follow-up is truncated at 7 years.
Prophylactic surgery based on trial data demands evidence that the outcomes achieved within a trial setting are reproduced in nontrial practice. The audit process must be transparent, with a degree of objectivity that satisfies the scrutiny of both professional and lay observers, regardless of whether this is a purely personal audit or audit conducted by, or on behalf of, a representative body. Similarly, the comparative auditing of individuals or institutions demands independent reproducibility with minimal scope for inter- and intraobserver error because of methodological differences.
Much time and effort is spent defining follow-up protocols for patients enrolled in RCTs. However this does not in itself confer absolute consistency of assessment,7 an issue further confused by a lack of consensus with respect to the details of assessment protocol between RCT.8–10,14 Furthermore, reproducing RCT standards of appropriately trained and independent assessment for every single carotid intervention is beyond most surgical units and the supporting stroke services.
To take advantage of routinely collected, but independently gathered, data for audit some license is required around the definition of stroke severity. Using readmission or intrahospital transfer as the criterion to define a clinically significant stroke relieves the operator of some of the burden of classifying the event severity, but this does imply a degree of acceptance that variation may occur around thresholds for readmission unrelated to the severity of the stroke, dependent instead on patient comorbidity, home support, and general practitioner, or patient preference. Furthermore, although surgeons and stroke physicians may choose to readmit and reinvestigate all patients developing new, focal neurological signs postoperatively, the present data indicate that on occasion general practitioners may not rerefer patients with a minor stroke for reassessment.
Of considerable interest to those seeking to interpret outcome audit of carotid surgery, Rothwell et al6 published an analysis of the published data in 1996; meta-analyses of outcome data were stratified according to the context and methods of data gathering. Variation was apparent between the rates of significant perioperative adverse events after carotid endarterectomy for symptomatic disease, dependent on follow-up protocol, independence of observer, and whether or not the data were recorded as part of a trial. This variation could be because of a variety of causes, not least of which is simple publication bias, but it does mean that data gathered within an RCT with stroke physician review probably represent the true evidence-based standard rather than personal series (even with independent review). Rothwell et al6 not only noted a relatively close agreement between the rates of death (and fatal stroke) but also observed a marked divergence between the reported rates of nonfatal stroke. Disparities between outcomes occur within current literature. For instance, the outcome data from the GALA trial (an RCT with independent stroke physician review) indicated that in the first 867 procedures, performed on symptomatic patients and recruited from the United Kingdom, there were 13 (1.5%) deaths and 45 (5.2%) stokes; 12 of the 45 strokes were fatal.5 However, the UK Carotid Endarterectomy Audit reporting data from contemporary carotid surgical practice in the United Kingdom indicated a combined stroke and death rate of just 2.5% of 4918 procedures in round 1 and 1.8% of 6135 procedures in round 2.15 Expressing some concern on nonvalidated outcome data, the authors stated data were self-reported, and it is possible that the morbidity and mortality figures under-represent the true complication rate. Highlighting the potential failings associated with a self-reporting system, the authors went on to recommend that independent postoperative review should become an essential part of the patient pathway. However as stated, this may not always be practical and would require significant resource.
Outcomes from major carotid intervention trials are summarized in Table 2.
A formal meta-analysis of these data has been published.23 Articles were selected if they contained data pertaining to surgical intervention, with a clearly stated indication being symptomatic disease and independent stroke physician review. The summated perioperative mortality is ≈1%, the majority of deaths are due to fatal stroke. The stroke rate was >5%. The ratios of the number of fatal to nonfatal perioperative strokes have been explored by other authors.6 Although there is a perception in recent years of improved outcomes after carotid intervention, the landmark RCTs still provide appropriate comparative data for analysis.23 The ECST reported 6 nonfatal strokes10 for each death and the North American Symptomatic Carotid Endarterectomy Trial reported 8.9
The outcome data presented seem consistent with those of the major trials in both number and nature of major complications. Because the number of deaths is low within this relatively small series, extrapolation to determine an expected number of nonfatal strokes must be considered unreliable. However, in the context of exploring the reliability of the gathered data, it is reassuring that the number of nonfatal strokes identified significantly exceeds the number of early (30 days) deaths. The ISD search was made on the basis of patients’ names, date of birth, CHI number, and date of procedure and it, therefore, seems unlikely that a death occurring within Scotland will have been missed. However, other important possible shortcomings of the present data must be considered. First, there are no comparative follow-up data from an independent source reviewing all the cases. Further prospective work will correct this shortcoming in due course and may identify a larger number of less severe strokes. Second, this process relies on a complete list of patients undergoing a procedure on which to base the follow-up search. In the past, surgical audit data may have been retrieved and recorded at the time of writing a discharge summary. However, it is likely that the notes of many patients with immediate or early complications will never return to the surgeon with the discharge or death summary being generated by a stroke team, perhaps in another hospital. Furthermore, any retrospective identification of cases is itself time-consuming and carries the risk of omission. The entry of data online to a centrally managed database just after or, perhaps more desirably, before the time of surgery, recording the nature and indication for the procedure, could serve to enhance transparency.
In the United Kingdom, data are collected from hospital discharge summaries and coding using ICD-10 or OPSC-4 codes. These are submitted to form the national statistics. It would be possible to request and analyze these data directly on each and every case without involving the surgical teams. However, the Audit Commission (a public corporation in the United Kingdom evaluating spending across public services) has recently addressed the quality of these data. It was estimated that incorrect clinical coding in England in 2010 cost Primary Care Trusts £1 billion.24 Although there may have been a subsequent reduction in coding errors,25 the accuracy is still less than the level acceptable to scrutinize any given individual’s work.
Accepting that complications are likely to occur, case selection is vital, and it is important to demonstrate that patients are living long enough to accrue benefit from the intervention. The present data put 5-year mortality at 18%, compared with 20% in NASCET.9 ECST reported deaths at a rate per month of 0.30%10; we report a rate of 0.32% per month. Death data are likely to be the most reliable for use as an end point and will indicate consistency and appropriateness of case selection between individuals, or at least individual units, in a large audit. Death data seem to select themselves as a starting point for quality assurance of data collection given that this outcome is the easiest to determine by any method. Data linkages allow confirmation that patients have been appropriately selected for surgery by confirming patient survival of sufficient duration to accrue benefit that justifies the operative risk.
The most time-consuming part of this process is maintaining a complete record of interventions. Around half of the readmissions after surgery were false-positive cases for postoperative stroke. Each case record was examined for documented evidence of a new stroke. Electronic records made this process more efficient for recent cases. The cases identified from earlier in the series required significant effort to gather evidence to diagnose stroke. We think that the time taken to perform this audit will be significantly reduced with each audit cycle.
The present data support the use of centrally held data to enhance and validate voluntary surgical audit. Providing that the initial, core data are accurate and complete, we propose that this process provides a pathway for returning to the operator a list of cases, which is likely to contain the majority of clinically significant adverse outcomes. It is our contention that making use of centrally held data can provide an accurate reflection of complication rates and confer a greater degree of objectivity and transparency on what remains in essence an externally ratified self-audit. The application of this methodology in different healthcare systems may not be immediately applicable (and indeed at present there may be legal implications to consider in terms of data protection). However, it would seem reasonable to anticipate that this will become easier as electronic patients records evolve, a process enhanced by a unique patient identifier (such as the CHI in Scotland, the personnummer in Sweden, or by social security numbers elsewhere in the world). It becomes imperative that healthcare management organizations be persuaded to allow analysts to develop these methods of interrogation to gather the information needed for follow-up.
Dr Ford acknowledges funding from the Wellcome Trust for the Scottish Health Informatics Program. The other authors have no conflicts to report.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.111.000200/-/DC1.
- Received November 18, 2012.
- Revision received February 20, 2013.
- Accepted February 22, 2013.
- © 2013 American Heart Association, Inc.
- Rerkasem K,
- Rothwell PM
- Dellagrammaticas D,
- Lewis S,
- Colam B,
- Rothwell PM,
- Warlow CP,
- Gough MJ
- Rothwell PM,
- Slattery J,
- Warlow CP
- Quinn TJ,
- Dawson J,
- Walters MR,
- Lees KR
- Farrell B,
- Godwin J,
- Richards S,
- Warlow C
- Rudarakanchana N,
- Halliday AW,
- Kamugasha D,
- Grant R,
- Waton S,
- Horrocks M,
- et al
- Wilkinson JM,
- Rochester JR,
- Sivaguru A,
- Cameron IC,
- Fisher R,
- Beard JD
- Bonati LH,
- Ederle J,
- McCabe DJ,
- Dobson J,
- Featherstone RL,
- Gaines PA,
- et al
- Ederle J,
- Dobson J,
- Featherstone RL,
- Bonati LH,
- ven der Worp HB,
- de Borst GJ,
- et al
- O’Dowd A
- 25.↵Audit Commission. Improving coding, costing and commissioning. Annual report on the payment by results data assurance programme 2010/2011. Available at http://www.audit-commission.gov.uk/SiteCollectionDocuments/Downloads/pbrannualreport2011.pdf. Accessed September 2011.