| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Stroke. 2008;39:3367.)
© 2008 American Heart Association, Inc.
Original Contributions |
From the Department of Neurology (A.K., C.B., R.G.H., J.P.T.) and the Office of Clinical Practice Evaluation (R.G.H., D.T.), University of Rochester, Rochester, NY.
Correspondence to Robert G. Holloway, MD, MPH, University of Rochester Medical Center, Department of Neurology, 601 Elmwood Avenue, Box 673, Rochester, NY 14642. Email Robert_Holloway{at}urmc.rochester.edu
| Abstract |
|---|
|
|
|---|
Methods— We performed a cross-sectional survey of all report cards within the Agency for Healthcare Research and Quality Report Card Compendium. Stroke quality data were categorized into one of 5 groups: structure, process, outcomes, utilization, and finances. We also determined the congruence of mortality ratings of New York hospitals provided by 2 different report cards.
Results— Of 221 available report cards, 19 (9%) reported quality information regarding stroke and 17 specifically addressed the quality of hospital-based stroke care. The most frequent data reported were utilization measures (n=15 report cards) and outcome measures (n=14 report cards). Data regarding finances (n=4), structure of care (n=2), and process of care (n=1) were reported infrequently. Ratings were incongruent in 61 of the 157 hospitals (39%) with the same hospital being rated below average on one report care and average on another in 44 hospitals.
Conclusions— Publicly reported quality data pertaining to patients with stroke are incomplete, confusing, and inaccurate. Without further improvements and a better understanding of the needs and limitations of the many stakeholders, targeted transparency policies for stroke care may lead to worse quality and large economic losses.
Key Words: mortality quality of care stroke
| Introduction |
|---|
|
|
|---|
Stroke care is of high priority on the national quality agenda. Outcomes measures are readily available (using administrative data), process measures have been developed from research evidence, and the Joint Commission and state health departments have encouraged the adoption of primary stroke centers.5–11 Quality initiatives have also shown that improvement on some of the measures is achievable.12,13 Despite these observations, the scope and accuracy of publicly reported quality data regarding stroke care are unknown.
In November 2006, the Agency for Healthcare Quality and Research (AHRQ) released its Report Card Compendium, which assembled all publicly available web sites reporting health quality data at a single Internet site.14 We reviewed the individual report cards to assess the content and scope of publicly available information regarding quality data after stroke. We also investigated the degree to which different sites provided similar estimates of quality regarding the same hospital.
| Materials and Methods |
|---|
|
|
|---|
Each quality measure was categorized as a measure of structure, process, outcome, utilization, or finance.16,17 Structural measures include the availability and quality of resources, management systems, and designation as a primary stroke center. Process measures (performance measure or effectiveness measures) evaluate the activities of physicians and other healthcare providers to determine if evidence-based recommendations are followed. Outcome measures evaluate the end result of health care and include measures of mortality, readmissions, complications, and patient/caregiver satisfaction scores. Utilization measures include data pertaining to the frequency of service use, including length of stay. Finance measures include economic data pertaining to the provision of stroke care.
We identified the rating of all New York state hospitals as provided by 2 separate report card sites: the Niagara Health Quality Coalition (NHQC), a not-for-profit corporation, and Health Grades, a for-profit, nationwide evaluation program.18,19 Both sites report on inpatient stroke mortality, use International Classification of Diseases, 9th Revision, Clinical Modification diagnosis codes to define their population at risk, and exclude patients transferred to another acute care hospital. Minor differences in coding rules exist between the 2 reporting systems (eg, Health Grades includes International Classification of Diseases, 9th Revision code 436 and excludes patients with a palliative care code, V66.7). Neither report card provided ratings of hospitals with low stroke volumes, which was defined as <30 cases per year. New York state was chosen as the source for this analysis due to the availability of 2 report cards, which separately evaluated nearly all hospitals statewide.
The NHQC accepts no advertising, consulting, or other funding from the providers it grades and uses statewide hospital administrative data (age 18 years and older) for the calendar year 2005. NHQC uses software developed by the AHRQ to compares each hospitals inpatient mortality with a risk-adjusted national average (AHRQ Inpatient Quality Indicators).6 This measure is risk-adjusted using a linear model estimated from a nationwide data set and includes age, gender, and All-Payer Refined Diagnostic Related Groups as developed by 3M.20 NHQC uses a 95% CI to identify which hospitals are better (3-star rating), worse (one-star rating), or not significantly different from the statewide average (2-star rating).
Health Grades uses Medicare inpatient billing data (age 65 years and older) for the period 2003 to 2006 and uses a proprietary "disease-specific and outcome-specific" risk-adjustment methodology using demographic characteristics, comorbid diagnoses, and specific procedures. Health Grades calculates a predicted mortality for each hospital using a proprietary risk-adjusted methodology, to which the actual hospital mortality is compared. The statistical methodology determining if the actual and predicted rates are significantly different is not readily available on the web site. Those hospitals whose mortality was lower than predicted were assigned 5-star ratings and those hospitals whose mortality was higher than predicted were assigned a one-star rating. Those hospitals whose actual performance was not significantly different from what was predicted received a 3-star rating (2-star and 4-star ratings are not assigned). According to methodology found on the companys web site, for each diagnosis or procedure, approximately 70% to 80% of hospitals should receive 3-star (average) ratings, whereas 10% to 15% of hospitals should receive one-star and 5-star ratings.
For each hospital in New York state that was included in both report cards, we assessed agreement between the 2 reporting systems for the following categorical ratings: (1) not significantly different than the state average or from what was predicted (ie, 2-star rating from NHQC and 3-star rating from Health Grades); (2) better than the state average or lower than predicted (ie, 3-star rating from NHQC and 5-star rating from Health Grades); and (3) worse than the state average or higher than predicted (ie, one-star rating from NHQC and one-star rating from Health Grades). We performed similar analyses for hospitals designated by the Joint Commission as primary stroke centers and hospitals designated by New York state as designated stroke centers.8,10 Weighted kappa was used to assess level of agreement.
This study was exempt from review by the University of Rochesters Research Subject Review Board.
| Results |
|---|
|
|
|---|
The 17 hospital-based report cards are listed in Table 1. The report cards were produced by state health departments,26,29,30,32,34,35 independent research organizations or private–public partnerships,18,25,27 insurance companies,22,23,33 for-profit companies,19,21 hospital associations,24,28 and health systems.31 All sites used hospital administrative data that were 1 to 3 years old, and 13 sites used the Inpatient Quality Indicator risk-adjustment software provided free by the AHRQ.6 One site created separate reports for nonhemorrhagic and hemorrhagic strokes.34 All other sites grouped subarachnoid hemorrhages, intracerebral hemorrhages, and ischemic strokes into one measurement cohort. Two sites included supplemental survey information about the structure and process of care.25,26
|
Table 2 shows the type of quality measures reported. The most frequently reported measures were utilization (n=15) and outcome (n=14) data. Risk-adjusted inpatient stroke mortality was the most commonly reported outcome measure. Two sites included quality indicators about the structure of stroke care services and only one site (from the United Kingdom) addressed the process of stroke care. Four sites included economic information with various methods of presentation (eg, costs, average charges, median charges).
|
The results of comparing inpatient mortality ratings for similar hospitals are summarized in Table 3. A total of 157 of 214 New York state hospitals were evaluated by both the NHQC and Health Grades (the majority of the 57 unrated hospitals were due to low volume). Health Grades rated 56 of the 157 hospitals below average compared with only 16 rated below average by NHQC. The 2 sites provided congruent ratings (average/average, below average/below average, above average/above average) on 96 of these 157 hospitals (agreement 61.1%, weighted kappa 0.163, slight agreement above that expected by chance). Ratings were incongruent in 61 of 157 cases (38.9%), including one case in which a hospital was rated above average by one site and below average by the other. The most common disagreement was an average rating by NHQC and a below average rating by Health Grades (44 of the 157 hospitals [28%]). Only one hospital was rated as above average by both sites. Agreement was similarly low for the 10 primary stroke centers (agreement 40%, weighted kappa 0.118) and the 107 of the 116 designated stroke centers evaluated by both sites (agreement 61.6%, weighted kappa 0.173).
|
| Discussion |
|---|
|
|
|---|
First, the data are incomplete. Despite there being well-established process measures for stroke, they were reported by only one site and this involved hospitals from the United Kingdom.8,9,11 Few sites reported on the structural elements of quality (stroke unit, accredited facility, designated stroke center), an easy potential addition given the published guidelines to establish both primary and comprehensive stroke centers.38,39 No sites reported on the quality dimensions of patient-centeredness (eg, patient satisfaction) or health disparities. One reason for this focus on outcomes and utilization quality reporting is the availability of administrative data, which does not generally include process or structure data.
Second, the data are poorly defined. The most common outcome measure reported is risk-adjusted in-hospital mortality rate, but it is not clear what this rate is actually measuring. Short-term mortality correlates poorly with process measures and is likely related to unsafe care in fewer than 10% of all deaths.40,41 In fact, the majority of stroke deaths occur after deliberate decisions by patients and their families not to pursue unwanted life-prolonging treatments.42 Short-term mortality, therefore, may be more indicative of "good quality" deaths, particularly because more informed patients are more inclined to want less aggressive care (ie, better quality decision-making leading to higher short-term mortality).43 The tremendous variability in how mortality and other outcome data are reported only compounds the confusion. It is also unclear if the average user knows how to interpret and use other measures that are frequently reported such as utilization data (eg, length of stay) or financial information (eg, charges versus costs).
Third, the data are unreliable. We found that 2 separate report cards provided disparate hospital ratings in 39% of comparisons. Disagreement was also observed among primary and designated stroke centers, a subset of hospitals selected for the capacity and quality of stroke care they provide. A recent study showed inconsistent ratings of hospitals among several sites for surgical procedures, but did not quantify the degree of disagreement.44 It is not clear why the report card ratings disagree so frequently. Potential reasons include different sample eligibility criteria, inconsistent methods of risk adjustment, and variable thresholds for defining statistical significant deviations from average or expected results. The potential for systematic bias should also be explored, particularly given the skew in below average ratings found in one of the report cards and their deviation from a predefined distribution of outlier status.
Unreliable and invalid publicly reported stroke quality data may have unintended consequences.3,4,43,45,46 Patients may choose the wrong providers, payers may reward or punish providers inappropriately, providers may "game" to improve rankings, hospital leaders may divert resources from worthy improvement efforts, and intermediary companies may profit by stoking fears of losing reputation and market share among affected hospitals. In the end, the public loses trust.
We provide 3 recommendations. First, efforts are needed to develop a standardized "dossier" of stroke quality measures that meaningfully align with the 6 worthy aims of health care: effective, safe, patient-centered, equitable, timely, and efficient.17 This objective will include efforts to harmonize existing stroke process measures (which are in progress) and to develop consensus metrics for stroke outcomes that measure "good-quality" deaths as well unexpected "never ever" deaths for which organizations should be held accountable.8,9,15,47,48 In addition, we need to develop and standardize new measures that focus on patient-centered, efficient, and equitable care. Collaborative public–private partnerships with several organizations that are currently committed to providing stroke quality data for internal quality improvement efforts could facilitate such efforts.9,49
Second, there should be more organized skepticism focused on the AHRQ stroke inpatient quality indicator as a primary measure of quality of care.6 The increasing appetite for healthcare quality data and the easy access of administrative data will likely guarantee the continued use of mortality as a marker of quality. In the short run, this will placate stakeholders. Fundamental questions remain, however, about the appropriateness of combining all types of stroke (subarachnoid hemorrhage, intracerebral hemorrhage, ischemic) into this one indicator and the impact that such measures may have on the delivery of high-quality palliative care. The inpatient time horizon is confounded by hospital practice patterns and the capacity of nonhospital services and ignores the longitudinal accountability needed to improve the quality of a chronic condition. Finally, despite its "public access," the risk-adjusted methodology remains proprietary.20
Third, further national efforts are needed to develop standardized reporting requirements with explicit rules to reduce bias and to ensure a minimum standard for measuring and reporting conduct.50 Much can be learned from the transparency systems that help govern corporate financing, restaurant hygiene, and mortgage-lending practices.51 As the quality field continues to mature, there will be increasing efforts to cherry-pick measures for marketing purposes. All measures should be reported, good or bad; there is no substitute to playing by the rules and working with integrity. Discussion is also needed regarding mandatory versus voluntary reporting, internal reporting with feedback versus public reporting, and how to finance a sustainable and effective transparency system that is responsive, interactive, and customized to stakeholder preferences and public concerns.
| Summary |
|---|
|
|
|---|
| Acknowledgments |
|---|
R.G.H. consults for Milliman Guidelines reviewing practice guidelines and Maximus, Inc.
Received February 28, 2008; revision received April 28, 2008; accepted May 5, 2008.
| References |
|---|
|
|
|---|
2. Steinbrook R. Public report cards—cardiac surgery and beyond. N Engl J Med. 2006; 355: 1847–1849.
3. Werner RM, Asch DA. The unintended consequences of publicly reporting quality information. JAMA. 2005; 293: 1239–1244.
4. Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet. 2004; 363: 1147–1154.[CrossRef][Medline] [Order article via Infotrieve]
5. Iezzoni LI, Shwarz M, Ash AS, Mackiernan YD. Predicting in-hospital mortality for stroke patients: results differ across severity-measurement methods. Med Decis Making. 1996; 16: 348–356.
6. Agency for Healthcare Research and Quality. AHRQ Quality Indicators. Available at: www.qualityindicators.ahrq.gov/. Accessed November 28, 2007.
7. Holloway RG, Vickrey BG, Benesch CG, Hinchey JA, Bieber J. Development of performance measures for acute ischemic stroke. Stroke. 2001; 32: 2058–2074.
8. The Joint Commission. Primary Stroke Center Certification. Available at: www.jointcommission.org/CertificationPrograms/PrimaryStrokeCenters. Accessed October 10, 2007.
9. American Heart Association/American Stroke Association. Get With the Guidelines—Stroke. Available at: www.strokeassociation.org/presenter.jhtml?identifier=3002728. Accessed October 10, 2007.
10. New York State Department of Health. NYS DOH Designated Stroke Centers. Available at: www.health.state.ny.us/nysdoh/ems/stroke/stroke.htm. Accessed November 28, 2007.
11. Centers for Medicare and Medicaid Services. Physician Quality Reporting Initiative (PQRI). Available at: www.cms.hhs.gov/pqri/. Accessed November 28, 2007.
12. Jacobs BS, Baker PL, Roychoudhury C, Mehta RH, Levine SR. Improved quality of stroke care for hospitalized Medicare beneficiaries in Michigan. Stroke. 2005; 36: 1227–1231.
13. LaBresh KA. Quality of acute stroke care improvement framework for the Paul Coverdell National Acute Stroke Registry: facilitating policy and system change at the hospital level. Am J Prev Med. 2006; 31 (suppl 2): S246–250.[CrossRef][Medline] [Order article via Infotrieve]
14. Agency for Healthcare Research and Quality. Report Card Compendium. Available at: www.talkingquality.gov/compendium. Accessed December 11, 2007.
15. Krumholz HM for the American Heart Association Quality of Care and Outcomes Research Interdisciplinary Writing Group. Standards for statistical models used for public reporting of health outcomes: an American Heart Association Scientific Statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group. Circulation. 2006; 113: 456–462.
16. Donabedian A. The quality of care. How can it be assessed? JAMA. 1998; 260: 1743–1748.
17. Institute of Medicine, Committee on Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
18. Niagara Health Quality Coalition/Alliance for Quality Healthcare. 2007 New York State Hospital Report Card. Available at: www.myhealthfinder.com. Accessed December 11, 2007.
19. Health Grades®. Free Hospital Rankings. Available at: www.healthgrades.com. Accessed December 11, 2007.
20. 3M. 3M APR-DRGs. Available at: www.3mhis2007.com/apr/. Accessed November 28, 2007.
21. About.com: Health. UCompare Health Care. Available at: www. ucomparehealthcare.com/hospital_start.html. Accessed December 11, 2007.
22. Blue Cross/Blue Shield of Minnesota. Healthcare Facts. Available at: www.healthcarefacts.org. Accessed December 11, 2007.
23. Blue Cross/Blue Shield of Tennessee/Tennessee Hospital Association. Hospital Quality Comparison. Available at: www.bcbst.com/tools. Accessed December 11, 2007.
24. Colorado Health and Hospital Association Performance and Quality Group. Colorado Hospital Quality. http://www.hospitalquality.org. Accessed December 11, 2007.
25. Dr Foster, Ltd Hospital Guide. Available at: www.drfoster.com/guides. Accessed December 11, 2007.
26. Florida Agency for Healthcare Administration. Florida Compare Care. Available at: www.floridacomparecare.gov. Accessed December 11, 2007.
27. Fraser Institute. Hospital Report Card: Ontario 2006. Available at: www.hospitalreportcards.ca/. Accessed December 11, 2007.
28. Kentucky Hospital Association. Kentucky Hospital Association Quality Data. Available at: www.kyha.com/QualityData/IQISite/default.htm. Accessed December 11, 2007.
29. Maryland Healthcare Commission. Hospital Guide. Available at: http://mhcc.maryland.gov. Accessed December 11, 2007.
30. Massachusetts Executive Office of Health and Human Services. Healthcare Quality and Cost Information. Available at: www.mass.gov/healthcareqc. Accessed December 11, 2007.
31. Norton Healthcare. Quality Report. Available at: www.nortonhealthcare.com/about/qualityreport. Accessed December 11, 2007.
32. Office for Oregon Health Policy and Research. Oregon Hospital Quality Indicators. Available at: http://egov.oregon.gov/DAS/OHPPR/HQ. Accessed December 11, 2007.
33. PacifiCare. Hospital Performance. Available at: http://pacificare.com. Accessed December 11, 2007.
34. Pennsylvania Healthcare Cost Containment Council. Interactive Hospital Performance Report. Available at: www.phc4.org/hpr. Accessed December 11, 2007.
35. Texas Healthcare Information Council/Texas Department of State Health Services. Indicators of Inpatient Care in Texas Hospitals. Available at: www.dshs.state.tx.us/thcic/publications/hospitals/HospitalReports.shtm. Accessed December 11, 2007.
36. National Committee for Quality Assurance. Recognized Physician Directory. Available at: http://recognition.ncqa.org/. Accessed December 11, 2007.
37. Missouri Department of Health and Senior Services. Managed Care Performance Monitoring. Available at: www.dhss.mo.gov/ManagedCare/Publications.html. Accessed December 11, 2007.
38. Alberts MJ for the Brain Attack Coalition. Recommendations for the establishment of primary stroke centers. Brain Attack Coalition. JAMA. 2000; 283: 3102–3109.
39. Alberts MJ for the Brain Attack Coalition. Recommendations for comprehensive stroke centers: a consensus statement from the Brain Attack Coalition. Stroke. 2005; 36: 1597–1616.
40. Peterson ED, Roe MT, Mulgund J, DeLong ER, Lytel BL, Brindis RG, Smith SC Jr, Pollack CV Jr, Neby LK, Harrington RA, Gibler WB, Ohman EM. Association between hospital process performance and outcomes among patients with acute coronary syndromes. JAMA. 2006; 295: 1912–1920.
41. Bradley EH, Herrin J, Elbel B, McNamara RL, Magid DJ, Nallamothu BK, Wang Y, Normand SL, Spertus JA, Krumholz HM. Hospital quality for acute myocardial infarction: correlation among process measures and relationship with short-term mortality. JAMA. 2006; 296: 72–78.
42. Jaren O, Selwa L. Causes of mortality on a university hospital neurology service. Neurologist. 2006; 12: 245–248.[CrossRef][Medline] [Order article via Infotrieve]
43. Holloway RG, Quill TE. Mortality as a measure of quality; implications for palliative and end-of-life care. JAMA. 2007; 298: 802–804.
44. Leonardi MJ, McGory ML, Ko CY. Publicly available hospital comparison web sites. Arch Surg. 2007; 142: 863–896.
45. Marshall MN, Shekelle PG, Leatherman S, Brook RH. The public release of performance data. What do we expect to gain? A review of the evidence. JAMA. 2000; 283: 1866–1874.
46. Marshall MN, Romano PS, Davies HTO. How do we maximize the impact of the public reporting of quality of care? Int J Qual Health Care. 2004; 16: i57–i63.[Abstract]
47. National Framework and Preferred Practices for Palliative and Hospice Care Quality: A Consensus Report. Washington, DC: National Quality Forum; 2006. Available at: www.qualityforum.org/pdf/reports/palliative/txPHreportPUBLIC01–29–07.pdf. Accessed May 22, 2007.
48. National Quality Forum. Serious Reportable Events in Healthcare 2006. Available at: www.qualityforum.org/projects/completed/sre/. Accessed May 22, 2007.
49. University HealthSystem Consortium. Available at: www.uhc.edu/. Accessed December 11, 2007.
50. Pronovost PJ, Miller M, Wachter RM. The GAAP in quality measurement and reporting. JAMA. 2007; 298: 1800–1802.
51. Fung A, Graham M, Weil D. Full Disclosure: The Perils and Promise of Transparency. New York: Cambridge University Press; 2007.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Stroke Home | Subscriptions | Archives | Feedback | Authors | Help | AHA Journals Home | Search Copyright © 2008 American Heart Association, Inc. All rights reserved. Unauthorized use prohibited. |