Challenges in Assessing Hospital-Level Stroke Mortality as a Quality Measure
Comparison of Ischemic, Intracerebral Hemorrhage, and Total Stroke Mortality Rates
Background and Purpose—Public reporting efforts currently profile hospitals based on overall stroke mortality rates, yet the “mix” of hemorrhagic and ischemic stroke cases may impact this rate.
Methods—Using the 2005 to 2006 New York state data, we examined the degree to which hospital stroke mortality rankings varied regarding ischemic versus hemorrhagic versus total stroke. Observed/expected ratio was calculated using the Agency for Healthcare Research and Quality Inpatient Quality Indicator software. The observed/expected ratio and outlier status based on stroke types across hospitals were examined using Pearson correlation coefficients (r) and weighted κ.
Results—Overall 30-day stroke mortality rates were 15.2% and varied from 11.3% for ischemic stroke and 37.3% for intracerebral hemorrhage. Hospital risk-adjusted ischemic stroke observed/expected ratio was weakly correlated with its own intracerebral hemorrhage observed/expected ratio (r=0.38). When examining hospital performance group (mortality better, worse, or no different than average), disagreement was observed in 35 of 81 hospitals (κ=0.23). Total stroke mortality observed/expected ratio and rankings were correlated with intracerebral hemorrhage (r=0.61 and κ=0.36) and ischemic stroke (r=0.94 and κ=0.71), but many hospitals still switched classification depending on mortality metrics. However, hospitals treating a higher percent of hemorrhagic stroke did not have a statistically significant higher total stroke mortality rate relative to those treating fewer hemorrhagic strokes.
Conclusions—Hospital stroke mortality ratings varied considerably depending on whether ischemic, hemorrhagic, or total stroke mortality rates were used. Public reporting of stroke mortality measures should consider providing risk-adjusted outcome on separate stroke types.
Public reporting efforts currently profile hospitals based on overall stroke mortality rates, yet the “mix” of hemorrhagic and ischemic stroke cases may impact this rate. Although ischemic and hemorrhagic stroke are commonly investigated under the broad term “stroke,” significant differences exist in prevalence, etiology, treatment, and outcomes. If a hospital's performance rating shifted depending on which mortality measures were used, then one should question the validity of the stroke mortality measure as a suitable metric for the comparison of hospital performance. Our objective was to examine the correlation in hospital performance if determined by total strokes versus ischemic versus hemorrhagic stroke outcomes and determine whether total stroke performance rankings are associated with the mix of ischemic versus hemorrhagic strokes.
Using the Agency for Healthcare Research and Quality Inpatient Quality Indicator Software (AHRQ IQI, Version 3.2),1 we reported hospital-level 30-day observed and expected stroke mortality rates, defined as death from any cause 30 days after the index stroke hospitalization, and ratios of observed to expected mortality rates (O/E) within the 2005 to 2006 New York Statewide Planning and Research Cooperative System data. Risk adjustment was performed by adjusting for age, gender, All Patient Refined-Diagnosis Related Groups (Version 24), and secondary diagnoses for comorbidity.1 Mortality after discharge was determined through the Social Security Administration Death Master File. Subarachnoid hemorrhage was excluded due to its low prevalence. To obtain a reliable estimate, we limited our analyses to hospitals with at least 25 intracerebral hemorrhages and ischemic strokes during the study period (40% hospitals in New York state).
Pearson correlation coefficients (r) were used to evaluate relationships among hospital-level O/E ratios obtained from intracerebral hemorrhage, ischemic, and total stroke. Hospitals were considered outliers if the 95% CI of O/E did not include 1 (mortality better—range <1; mortality worse—range >1; or no different than average—range includes 1). Level of agreement for all 3 measures was quantified using weighed κ. The ratio of intracerebral hemorrhage to ischemic stroke case volume for each hospital was calculated to represent the mix of stroke types. This ratio was then divided into quartiles of increasing number of intracerebral hemorrhage to determine whether total stroke performance rankings were associated with the mix of strokes cases. All tests were evaluated at a 2-sided significance level of P<0.05. All statistical analyses were performed using SAS 9.2 (SAS Institute, Cary, NC).
Our analysis cohort comprised 31 629 records (26 218 ischemic and 5411 intracerebral hemorrhage) from 81 hospitals in New York state. The average 30-day stroke mortality rate was 15.2% and varied from 11.3% for ischemic stroke and 37.3% for intracerebral hemorrhage.
Hospital risk-adjusted ischemic stroke O/E ratio was weakly correlated with its own intracerebral hemorrhage O/E ratio (Figure 1; r=0.38). Hospital performance ratings comparing ischemic versus intracerebral hemorrhage (mortality better, worse, or no different than average) were misclassified in 35 of 81 hospitals (43.2%) with poor level of agreement (Table; κ=0.23). Agreement was similarly low (κ=0.36 and r=0.61) between intracerebral hemorrhage and total stroke (misclassified in 28 of 81 hospitals). The level of agreement improved (κ=0.71, r=0.94) when comparing ischemic and total stroke mortality ratings. However, 17 hospitals (21%) still switched performance groups (Table). Hospital-specific O/E ratios and 95% CI for all 3 measures are listed in the online-only Data Supplement Appendix.
Hospital hemorrhagic to ischemic stroke case volume ratios varied from 0.07 to 0.54 (median, 0.19; interquartile range, 0.15–0.25). Only 5% of hospitals in the highest quartile (Quartile 4) with relatively more hemorrhagic stroke had mortality worse than average, 60% had rates that were no different, and 35% had mortality better than average (Figure 2). In comparison, 35% of hospitals in the lowest quartile (Quartile 1) with fewer hemorrhagic strokes had risk-adjusted total stroke mortality rates worse than average, 35% had rates that were no different, and 30% had rates better than average. However, the difference among each quartile in terms of hospital performance was not statistically significant (P=0.20).
Total stroke mortality, although readily assessable from hospital administrative data, is inherently difficult to measure in a way that allows fair comparisons among providers due to the mix of stroke cases. Using a large state sample of stroke admissions, we assessed the degree to which hospital stroke mortality rankings varied whether one assessed hemorrhagic versus ischemic versus total stroke outcomes.
We found a high degree of variability in the composition of stroke types across hospitals with a hemorrhagic to ischemic case volume ratio from 0.07 to 0.54. This imbalance in the stroke population makes precise mortality estimates difficult. Importantly, hospital mortality rankings varied considerably depending on whether ischemic, hemorrhagic, or total stroke mortality rates were used. Poor agreement was observed between ischemic stroke and intracerebral hemorrhage mortality ratings (κ=0.23). Although the level of agreement improved (κ=0.71) when comparing ischemic and combined total stroke, disagreement still existed in >20% of the hospitals. This raises concerns over using total stroke mortality for hospital profiling, because inappropriate combining different stroke types may markedly affect which hospitals were identified as superior or inferior performers. Hospitals with a lower proportion of hemorrhagic stroke are more likely to have worse than average total stroke mortality rates. This contradicts our expectation that hospitals may be penalized by having more seriously ill patients. Possible explanations include “practice makes perfect” representing better performance in high-volume centers, overadjustment for intracerebral hemorrhage in the model, or different end-of-life treatment practices across hospitals.2,3
Given the growing interest in public reporting of hospital performance and upcoming stroke mortality measure required by the Centers for Medicare & Medicaid Services,4,5 there is an urgent need to develop standard stroke performance metrics. Future public reporting efforts should consider providing feedback on separate stroke types in addition to total stroke mortality. Although not currently collected by the Centers for Medicare & Medicaid Services, the inclusion of the National Institutes of Health Stroke Scale could provide quantitative information on stroke severity and better modeling predictors of stroke mortality.6 Finally, mortality is an important quality measure but by no means the sole quality measure. Inclusion of structural measures such as procedural volume and process measures such as adherence to guideline recommendations as well as other outcome measures including functional status, quality of life, and patient satisfaction could provide a more accurate and comprehensive measure of hospital quality of stroke care.
Our study has limitations. First, the AHRQ IQIs use the All Patient Refined Diagnosis Related Groups (APR-DRGs) and secondary diagnoses for comorbidity risk adjustment based on hospital administrative database. Details of the risk adjustment methodology remain proprietary and are not represented in the public domain. The validity of this method and quality of such data can be limited. Second, we excluded subarachnoid hemorrhage due to its low prevalence. We also excluded hospitals with <25 intracerebral hemorrhage/ischemic stroke cases to obtain a reliable estimate. Hence, our results might not be representative of acute stroke care in small hospitals. Finally, it should be emphasized that the results obtained from the AHRQ IQIs may not be generalizable to other public reporting tools with different risk adjustment metrics.
In conclusion, our findings suggest that combined total stroke mortality rates may not accurately represent hospital-level quality of stroke care due to the mix of stroke types. Future public reporting efforts should consider providing volume, stroke care process as well as risk-adjusted subtype specific stroke mortality measures.
Sources of Funding
This study received infrastructure support from the Agency for Healthcare Research and Quality (U18HS016964) and an award from the American Heart Association–Pharmaceutical Roundtable and David and Stevie Spina.
R.G.H. is a consultant for Milliman Guidelines reviewing neurology guidelines.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.111.648600/-/DC1.
- Received January 23, 2012.
- Revision received March 12, 2012.
- Accepted March 13, 2012.
- © 2012 American Heart Association, Inc.
AHRQ Inpatient Quality Indicators Overview. Available at: www.qualityindicators.ahrq.gov/Modules/iqi_overview.aspx. Accessed April 11, 2010.
Hospital Compare. US Department of Health & Human Services. Available at: www.hospitalcompare.hhs.gov/. Accessed May 21, 2010.
- Mackey J,
- Kleindorfer D
- Smith EE,
- Shobha N,
- Dai D,
- Olson DM,
- Reeves MJ,
- Saver JL,
- et al