Evidence-Based Quality Indicators for Stroke Rehabilitation
Background and Purpose—Previous stroke performance measures consider aspects of postacute treatment, but there are only few specific quality indicators or standards for poststroke rehabilitation. The purpose of this study was to develop a set of indicators for measuring the quality of postacute stroke rehabilitation in inpatient and outpatient facilities using a standardized evidence-based approach.
Methods—Quality indicators were developed between January 2009 and February 2010 by an interdisciplinary board of healthcare professionals from rehabilitation centers cooperating in the Berlin Stroke Alliance. The Berlin Stroke Alliance is a regional network of >40 providers of acute treatment, rehabilitation, and aftercare aiming to improve stroke services within Berlin and Brandenburg. The indicators were developed according to published international recommendations and predefined methodological requirements. The applied standards included a systematic literature review, a rating of published evidence, an external peer review, and the evaluation in a pilot study before implementation.
Results—Of an initial list of 33 indicators, 20 indicators were rated as being appropriate. After completion of the pilot phase, we agreed on a set of 18 indicators. The indicators measure processes (9 indicators), outcomes (5 indicators), and structures (4 indicators) in the following domains of stroke rehabilitation: completion of diagnostics; secondary prevention; cognition and affect; speech and swallowing; management of complications; sensorimotor functions and mobility; discharge status; and aftercare.
Conclusions—Documentation of evidence-based quality indicators for stroke rehabilitation in clinical routine is feasible and can serve as a first step toward implementing standardized cross-institutional quality assurance programs for stroke rehabilitation.
Treatment approaches that have proved to be effective within the settings of clinical trials must be implemented in routine clinical practice to have a positive impact on the health of the population. A widely accepted procedure for translating research evidence into clinical practice is to monitor quality of care using standardized performance measures (the terms quality indicators and performance measures are used synonymously).1
Quality indicators can be used to evaluate the adherence to current guidelines. They can both enforce and accelerate the transfer of new scientific evidence into everyday clinical practice.2
An increasing number of quality initiatives for acute stroke care have recently been established in Europe and the United States.3–8 Ensuring high standards of care, however, is important not only for acute treatment, but also for rehabilitation efforts in later stages of stroke care.
There is evidence of the efficacy of postacute rehabilitation in reducing mortality and dependency of stroke patients.9–11 Greater adherence to published rehabilitation guidelines has been demonstrated to be significantly linked with improved functional outcomes of patients with stroke.12,13
Despite these figures, so far, there are only few quality indicators for stroke rehabilitation14 and only a limited number of specific standards for postacute rehabilitation of patients with stroke such as the standards of the Stroke Speciality Program of the Commission of Accreditation of Rehabilitation Facilities.15 Therefore, we decided to develop a set of indicators that allow measuring the performance of inpatient and outpatient stroke rehabilitation. We used a standardized evidence-based approach to develop these indicators and tested their feasibility in clinical routine in a pilot study. The following summary reports the theoretical background, the methods applied to develop quality indicators for stroke rehabilitation, and the results of the pilot study.
Constitution of the Consensus Group
The Berlin Stroke Alliance (BSA) is a regional network of >40 providers of acute treatment, rehabilitation, and aftercare of patients with stroke established to improve stroke services in Berlin and Brandenburg. In 2009, the BSA initiated an interdisciplinary board to develop evidence-based quality indicators for stroke rehabilitation. Representatives with long-term working experience from inpatient as well as outpatient rehabilitation institutions were involved in the development process. The majority of the board members were physicians holding either specialist training in physical and rehabilitative medicine or in different medical disciplines (eg, neurology, geriatrics, internal medicine) with approximately half of them having an additional certified qualification in rehabilitation medicine. The group also included a sociologist and a rehabilitation psychologist. The working group members discussed the developed proposals within the multidisciplinary team of the respective facility and fed back their comments to the whole group.
The standardized approach chosen by the working group members to develop quality indicators was based on the recommendations of the “First Scientific Forum on Assessment of Quality of Care and Outcomes Research in Cardiovascular Disease and Stroke” of the American Heart Association2 and the requirements for clinical performance measures for use in the German healthcare system.16 These included a systematic literature review, a rating of published evidence, and an external peer review. A similar approach to develop quality indicators for acute stroke treatment was recently chosen by the German Stroke Registers Study Group (ADSR), a voluntary network of regional stroke registers for monitoring quality of acute stroke care.4
Methodological Requirements of Quality Indicators
Based on previous recommendations, the following methodological requirements for quality indicators were agreed on by the working group2,16: the indicators should be based on the best scientific evidence available. Outcomes measured should be meaningful for the patients or closely linked to relevant outcomes. Measures should be reliable and allow for case-mix adjustment of the participating institutions to ensure that observed differences are related to performance differences rather than disparities in patient characteristics. Furthermore, they should be sensitive to changes in the provision of care to encourage healthcare providers to improve their services. It should be feasible for staff members of the participating rehabilitation centers to collect the data and the effort required for the data collection should be kept at a minimal level.
Dimensions of Quality and Components of Stroke Rehabilitation
The working group chose to follow the Donabedian concept, which differentiates among structure, process, and outcome as dimensions of healthcare quality.17 Structure refers to features and resources of a given institution. Process refers to the use of appropriate therapeutic and diagnostic procedures. Outcome indicators assess the influence of healthcare delivery on the individual patient's health.2
In addition, the working group defined the following major components of stroke rehabilitation to be covered by the set of indicators: completion of diagnostics, secondary prevention, cognition and affect, speech and swallowing, management of complications, sensorimotor functions and mobility, and discharge status and aftercare.
The 3 dimensions of healthcare quality and the 7 components of stroke rehabilitation constituted a framework for defining possible quality indicators without either overemphasizing or neglecting certain aspects of care. For each of the components of stroke rehabilitation, the working group defined a set of possible indicators representing structures, processes, and outcomes of care.
Evaluating the Quality of Scientific Evidence
For each possible indicator, a literature search was performed using standardized search algorithms in PubMed and the Cochrane Database of Systematic Reviews. In addition, guideline databases and Web sites of neurological associations were searched for guidelines for stroke treatment. The results of this research were presented to the working group in standardized reports for each indicator. We gave priority to studies with a higher level of evidence and higher methodological quality. If there was no scientific evidence available, consensus or expert statements were taken into account instead. Clinical guidelines were evaluated with a standardized tool for the assessment of methodological guideline quality recently proposed in Germany (Deutsches Instrument zur methodischen Leitlinien-Bewertung [DELBI]) before their recommendations were considered in the reports.18
Scientific evidence was graded using an adapted version of the criteria issued by the Oxford Centre for Evidence-Based Medicine.19
Specification and Rating of the Indicators
To decide on a final set of indicators, all potential indicators were rated by the members of the consensus group on a 5-point Likert scale from 1 (definitely not applicable) to 5 (definitely applicable) taking into account the predefined methodological requirements and the strength of scientific evidence. Only indicators that scored at least 4 of 5 points on this scale were included in the final set of indicators.
Indicators related to processes and outcomes were defined as proportions of populations with accurately defined numerators and denominators. The numerator includes all patients who receive the intervention of interest or who show the defined outcome, whereas the denominator comprises all patients who are expected to benefit from the intervention or who are at risk to show the considered outcome.
Subsequently, the indicators underwent an external review by an independent expert in neurological rehabilitation for completeness and meaningfulness of the chosen set.
The study was approved by the ethics committee of the Charité–Universitätsmedizin Berlin (EA1/170/10). The data management concept was positively evaluated by the corresponding data protection officer.
Development of a Preliminary Set of Quality Indicators
The development of quality indicators for stroke rehabilitation took place during 8 workshops from January 2009 until February 2010. A detailed time table of the process is presented in Supplemental Table I (http://stroke.ahajournals.org). Using the framework of predefined components of stroke rehabilitation and dimensions of health care, an initial list of 33 possible indicators was generated. A standardized literature research was performed for each of the potential indicators. Of 4790 titles identified in PubMed and the Cochrane Database of Systematic Reviews, approximately 800 abstracts were screened and 134 publications were analyzed in detail. Twenty-nine guidelines were identified of which 11 guidelines were considered in the reports. The selection of the final set of indicators took place during 3 working group meetings. Of the 33 potential indicators, 20 indicators were rated appropriate. The initial set along with the final set of indicators is presented in Supplemental Table IV.
Ten of these 20 indicators related to processes, 6 indicators related to outcomes, and 4 indicators related to structural characteristics of the institution. An independent expert in stroke rehabilitation not previously involved in the development process was asked to review the development process and the set of indicators. He was provided with a report including a description of methodological proceedings and a detailed presentation of the indicators. An example of how the indicators were presented is shown in Supplemental Table IV. The overall judgment of the indicators was positive; remarks were mostly related to the presentation of the indicators and led only to minor modifications.
A pilot study to test the feasibility of the indicators related to processes and outcomes and the variations in performance was conducted in 3 rehabilitation facilities of the BSA (1 inpatient neurological unit, 1 inpatient geriatric unit, and 1 outpatient neurological unit). All patients with a diagnosis of stroke (International Classification of Diseases 10 G45, I60, I61, I63, I64) who were admitted between September 2010 and November 2010 were included in the study. A case report form listing variables necessary for calculating the indicators was drafted. The form also comprised inclusion and exclusion criteria and potential case-mix variables. A standardized manual of operation was developed and served as a guide to the data collection. We also contacted the 3 rehabilitation facilities to inquire about the indicators related to institutional structures. During the recruitment period, 162 patients were included in the pilot study. The mean age of the patients was 68 years (SD, 14); 40% of the patients were females. Eighty-six percent of the patients had cerebral infarction, 6% intracerebral hemorrhage, and 3% subarachnoid hemorrhage. The mean Barthel Index on admission to rehabilitation was 79 out of a 100-point scale (SD, 31); the mean time between the onset of symptoms and admission to a rehabilitation facility was 5 weeks (SD, 9 weeks); the average length of stay in a rehabilitation center was 5 weeks (SD, 2 weeks). Average performance for the different indicators and variance between the participating centers vary widely (Table).
Based on the results of the pilot study, it was decided to discard 2 of the indicators due to a very small population in the denominator (“carotid artery imaging in patients with ischemic stroke/transient ischemic attack”) and to very few outcome events (“percentage of patients with documented falls during rehabilitation”).
It was further agreed to compare the results of the indicators measuring outcomes such as recovery of walking and upper limb function stratified by the degree of the patients' functional deficits using the phases of neurological rehabilitation (B, C, and D) as defined by the German “Federal Working group on Rehabilitation.”20
Feedback from the participating centers on feasibility and content of the case report form and the manual was mainly positive and the items were rated comprehensible and sensible. The average time to complete the form for 1 patient was 3 to 5 minutes. The final set of quality indicators is shown in Supplemental Table IV.
The case report form and the manual have been revised according to the results of the pilot study. The attending rehabilitation facilities agreed to implement the indicator set in the rehabilitation centers of the BSA starting from September 2011. Standardized feedback of performance will be given to the participating centers at regular intervals. Regular audits are intended to ensure correct data documentation.
This report describes the development of quality indicators to measure the performance in postacute stroke rehabilitation by a consortium of in and outpatient rehabilitation centers in Berlin and Brandenburg. The working group presented a final set of 18 indicators related to processes (9 indicators), outcomes (5 indicators), and structures (4 indicators) of stroke rehabilitation. The indicators were developed and evaluated according to published evidence and predefined methodological requirements. They underwent an external review by an independent expert in stroke rehabilitation and have been tested in a prospective pilot study that demonstrated their feasibility before being implemented in the participating centers.
Other stroke performance measures are mostly limited to acute hospital care, whereas indicators focusing on stroke rehabilitation are currently lacking.3–8 Some indicator projects, for example, in Denmark21 and Finland,22 however, have included follow-up surveys to evaluate long-term outcomes of patients with stroke as well. Furthermore, there are initiatives that have widened their scope by also covering ambulatory settings, for instance, the Swedish Stroke Register Riks-Stroke.23
A recently published review that analyzed commonly cited process indicators for stroke care criticized a medical bias in most of the indicator sets with very few indicators evaluating nonmedical aspects of care.24 The indicators presented here display a broader range of health care, including neuropsychology, speech therapy, physiotherapy, social work, and nursing care. We accounted for those aspects of care by defining a priori important components of stroke rehabilitation, which should be covered by the indicators. Therefore, the indicator set presented in this article stands out among other performance measures for stroke care.
The majority of the indicators measure processes of care, which are expected to translate into patient outcomes. Assessing processes that are closely linked to meaningful outcomes is often preferred to observing outcomes mainly due to 2 reasons. First, the comparison of outcomes requires risk adjustment for patient variability between the participating centers, which often lacks completeness because relevant confounders may be either unknown or immeasurable. Incomplete risk-adjustment strategies severely reduce comparability of performance. Second, process measures are often more readily accepted by providers and clinicians because they state clearly how to reach performance improvements. However, they must be constantly updated to correspond to current knowledge.25
Our indicator set also includes measures that are related to outcomes. These measures mainly focus on the restoration of functional deficits. This is due to the fact that these are the ones most likely to be influenced by the quality of postacute rehabilitation.
Our indicator set can facilitate the transfer from scientific evidence into clinical routine. It will thus extend monitoring efforts in Germany already established in acute stroke care to the rehabilitation sector. To meet these demands, the indicators need to be constantly evaluated and updated to make sure they correspond with the current state of research. An enhanced interdisciplinary dialogue on quality of care both between and within the participating centers as well as feedback, which actively involves the participating clinicians, will help to initiate changes in care delivery.1,26 If successful in improving quality of stroke care in the participating centers in and around Berlin, the project might also lead to progress in other regional networks of stroke rehabilitation.
There are several limitations of the study. First, the evidence base for different components of stroke rehabilitation varies considerably. Therefore, a significant proportion of the developed indicators is based on a rather low level of evidence, on extrapolations of related research findings from acute care settings, or on expert opinions. This is the case especially for the indicators related to social counseling issues, but also for interventions concerning cognitive, affective, or speech deficits of patients with stroke. This reflects the state of evidence-based medicine in the rehabilitation sector and clearly demonstrates the need for clinical trials within the setting of stroke rehabilitation. There is still further need to develop indicators related to cognitive and affective interventions that are suitable for aphasic patients. However, we may not have identified all relevant studies because we did not perform an independent literature research by a second rater. Second, our working group included rehabilitation facilities in Berlin and its surrounding areas only; national medical societies have not been involved, which in turn could impede or complicate widespread acceptance. The fact that we did not formally request the participating facilities to dispatch any members of nonmedical disciplines (ie, social workers or speech therapists) at the time of establishing the working group is clearly a limitation to our approach. Third, rehabilitation and acute care in Germany are offered in separate units due to the high fragmentation of German health care. Therefore, the indicators are of limited transferability to other healthcare systems. However, the indicators might help to identify areas that are underrepresented in indicator sets currently implemented in other healthcare systems. Finally, further evidence that better adherence to indicators measuring processes of rehabilitation care is linked to improved outcomes is still needed.
The development of quality indicators for postacute rehabilitation of patients with stroke addresses an existing gap in quality assurance of stroke treatment in Germany. Definition of quality indicators has been a first step toward implementing cross-institutional quality assurance programs. Our project may serve as an example for initiating similar national or international initiatives for measuring and improving quality of stroke rehabilitation.
Sources of Funding
This study was supported by the Federal Ministry of Education and Research (BMBF) through the Center for Stroke Research Berlin (01 EO 0801).
P.U.H. receives research support from the German Ministry of Education an Research (BMBF) within the Center for Stroke Research Berlin.
We wish to thank Prof Dr Christian Dettmers, Kliniken Schmieder Konstanz, for his support in reviewing the indicators. Furthermore, we wish to thank the healthcare professionals of Median Klinik Berlin-Kladow, Median Klinik Berlin-Mitte, and Vivantes Rehabilitation GmbH for their support in the pilot study.
The online-only Data Supplement is available at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.111.627679/-/DC1.
Coinvestigators: Charité–Universitätsmedizin Berlin: Andreas Meisel, MD, and Meike Sieveking; Median Klinik Berlin-Kladow: Anne-Kathrin zur-Horst-Meyer, MD; Median Klinik Berlin-Mitte: Nidal Mansour, MD.
- Received May 30, 2011.
- Revision received August 11, 2011.
- Accepted August 23, 2011.
- © 2012 American Heart Association, Inc.
Measuring and improving quality of care: a report from the American Heart Association/American College of Cardiology first scientific forum on assessment of healthcare quality in cardiovascular disease and stroke. Circulation. 2000; 101: 1483– 1493.
- Heuschmann PU,
- Biegler MK,
- Busse O,
- Elsner S,
- Grau A,
- Hasenbein U,
- et al
- Holloway RG,
- Vickrey BG,
- Benesch C,
- Hinchey JA,
- Bieber J
- Abilleira S,
- Gallofre M,
- Ribera A,
- Sanchez E,
- Tresserras R
- Cumbler E,
- Murphy P,
- Jones WJ,
- Wald HL,
- Kutner JS,
- Smith DB
- Reeves MJ,
- Parker C,
- Fonarow GC,
- Smith EE,
- Schwamm LH
- Duncan PW,
- Horner RD,
- Reker DM,
- Samsa GP,
- Hoenig H,
- Hamilton B,
- et al
Commission on Accreditation of Rehabilitation Facilities. Medical rehabilitation. Available at: http://carf.org/Programs/Medical/. Accessed August 2011.
- Geraedts M,
- Selbmann HK,
- Ollenschlaeger G
- Donabedian A
Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e.V. (AWMF); Ärztliches Zentrum für Qualität in der Medizin (ÄZQ). Deutsches Instrument zur methodischen Leitlinien-Bewertung (DELBI), Fassung 2005/2006 + domäne 8. 2008. Available at: www.versorgungsleitlinien.de/methodik/delbi/pdf/delbi05.pdf. Accessed June 2011.
Oxford Centre for Evidence-based Medicine (CEBM). Levels of Evidence. 2009. Available at: www.cebm.net/index.aspx?o=1025. Accessed June 2011.
Bundesarbeitsgemeinschaft für Rehabilitation. Empfehlungen zur Neurologischen Rehabilitation von Patienten mit schwersten Hirnschädigungen in den Phasen C und D. 1995. Available at: www.bar-frankfurt.de/upload/Rahmenempfehlung_neuroligische_Reha_Phasen_B_und_C_149.pdf. Accessed June 2011.
- Mainz J,
- Krog BR,
- Bjornshave B,
- Bartels P
- Meretoja A,
- Roine RO,
- Kaste M,
- Linna M,
- Juntunen M,
- Erila T,
- et al
- Rubin HR,
- Pronovost P,
- Diette GB
- Helou A,
- Schwartz FW,
- Ollenschlaeger G