Development and Implementation of Evidence-Based Indicators for Measuring Quality of Acute Stroke Care
The Quality Indicator Board of the German Stroke Registers Study Group (ADSR)
Background and Purpose— There is no consensus about indicators for measuring quality of acute stroke care in Germany. Therefore, a standardized process was initiated recently to develop and implement evidence-based indicators for the measurement of quality of acute hospital stroke care.
Methods— Quality indicators were developed by a multidisciplinary board between November 2003 and December 2005. The process was initiated by the German Stroke Registers Study Group in cooperation with the German Stroke Society, the German Society of Neurology, the German Stroke Foundation, Regional Offices for Quality Assurance and other experts proven in the field. National and international recommendations were considered during the development process. The process was based on a systematic literature review, an independent external evaluation of the process and its results, and a prospective pilot study to evaluate the defined indicators in clinical practice.
Results— Overall a set of 24 indicators was developed to measure performance of acute care hospitals in the 3 health care dimensions structure, process and outcome as well as in 3 treatment phases prehospital, in-hospital/acute and postacute. Practicability of the derived indicators was tested in a prospective pilot study. During a 2-month period, 1006 patients in 13 hospitals were documented. Application of the new indicator set was found to be feasible by participating physicians and hospitals. Median time to document the required information for 1 patient was 5 minutes. Nationwide implementation of the new indicator set within regional registers in Germany started since April 2006.
Conclusions— The development of indicators to measure hospital performance in stroke care is an important step toward improving stroke care on a national level. The chosen standardized evidence-based approach ensures maximal transparency, acceptance and sustainability of the developed indicators in Germany.
Convincing evidence is available that effective acute management and treatment, as well as adequate secondary prevention, reduce morbidity and mortality after a stroke event had occurred.1–4 In this context it has to be ensured that the evidence, mainly derived from clinical trials, influences routine clinical care on the community level.5,6 Thus, guidelines and consensus statements recommend the implementation of systems for systematically monitoring the quality of acute stroke care in the community.3,7 In several countries initiatives to measure quality of stroke care had recently been started: for example, the recommendations for quantifying healthcare quality in cardiovascular disease and stroke.8 The development of stroke-specific certification programs for hospitals.9 Stroke Unit–specific certification procedures.10 or the implementation of external audits to describe national standards of stroke care.11–14 Quality indicators are a widely used tool within programs for the improvement of quality of acute care. They measure performance of an individual facility over time, compare quality of care between different health care providers, and can identify areas for improvement.15 Quality indicators are understood as standards of care for all appropriate patients and must be developed in a standardized, evidence-based way.8,16
Up to now there is no national consensus about adequate indicators for measuring quality of stroke care in Germany. Therefore, a standardized evidence-based process was established to develop these indicators. The following summary reports theoretical background, used methodology, results and implementation of performance measures for stroke care.
Constitution of the Quality Indicator Board
Regional hospital-based stroke registers have been established in different federal states and districts in Germany since 1994. This implementation was based mainly on agreements between healthcare providers, health insurance companies and medical authorities, eg, boards of physicians. Main aim of these registers is a continuous monitoring of quality of acute stroke care. Academic and community hospitals within defined regions prospectively document data from consecutively admitted stroke patients in a standardized way.17 Predefined processes and outcomes are benchmarked between participating hospitals. The German Stroke Registers Study Group (ADSR) is a voluntary network of these ongoing regional registers.
For the development of evidence-based quality indicators in Germany, all professions and organizations involved in acute stroke care, stroke societies, quality improvement organizations as well as patient organizations in Germany were invited to join the Quality Indicator Board of the ADSR. Representatives of the Executive Board and the Commissions ‘Stroke Unit’ and ‘Health Care Research’ of the German Stroke Society, the Commission ‘Quality Management’ of the German Society of Neurology, the German Stroke Foundation, Regional Offices for Quality Assurance, as well as specialists from the disciplines internal and geriatric medicine, and other experts from the field of quality management in acute stroke care agreed to join the Quality Indicator Board.
Methodological Procedures and Definitions
The standardized process for developing quality indicators was initialized within a workshop in November 2003 and ended after 4 additional workshops in December 2005. During the first meeting, methodological procedures to define quality indicators were agreed on. For this purpose published national and international recommendations for developing indicators of quality of health care were reviewed and presented in a standardized way.8,18,19 In addition, experiences from research groups actually developing performance measures for stroke were reported.11,16,20 The board decided to use the recommendations of the First Scientific Forum on Assessment of Quality of Care and Outcomes Research in Cardiovascular Disease and Stroke8 and the Standards for Defining Clinical Performance Measures in Germany18 as the basis of this development.
Definition of the Term Quality Indicator
The following definitions of the term quality indicator were used8,18: “quality indicators” are explicit standards of care against which actual clinical practice is judged; quality indicators should be followed for all suitable patients with the exception of extraordinary circumstances; quality indicators must also define how to practically identify those patients for whom a specific action should be taken; evidence-based guidelines can be used as a basis for their development but quality indicators are no guidelines.
Dimensions of Health Care to be Covered
The Donabedian concept was chosen as principle basis for the dimensions of acute stroke care to be covered by indicators.21 According to this concept, healthcare quality can be separated into structure, process and outcome.21 To consider relevance of different healthcare domains more carefully,8 the board decided to add the different phases prehospital, in-hospital/acute and postacute stroke care to the Donabedian concept, yielding in total 9 components of stroke care to be covered by indicators. These components served as a structural framework for selecting performance measures. For each dimension of health care, indicators were aimed to be defined to provide a profile of indicators without overemphasizing specific components.18
Methodological Requirements of Quality Indicators
Methodological requirements for ensuring validity of quality indicators were defined as follows8,18: a quality indicator must be meaningful; any potential indicator must be either a meaningful outcome to patients and society or closely related to such an outcome; the indicator must be valid and reliable; to serve as a useful marker of healthcare quality, it must be possible to measure structure, process, or outcome of interest; the indicator must be adjustable for patient variability so that observed differences between hospitals are attributed to the hospital performance and not caused by differences in patient characteristics; the indicator can be modified by improvements in processes of care, requiring variability after risk adjustment among hospitals; measuring performance of healthcare providers must be feasible on a routine basis.
Presentation of Quality Indicators
Following a proposal of the Joint Commission on Accreditation of Healthcare Organizations9 it was decided that a standardized report for each quality indicator should be provided. The report included: component of healthcare quality to be covered; rationale for selecting the indicator including references (original publications, guidelines and consensus statements); definition of suitable patients for whom the quality indicator is valid (included and excluded population for numerator and denominator); possible adjustment for patient variability; way of data report (eg, rate or proportion).
Selection of Quality Indicators
The board agreed to define potential indicators for measuring quality of stroke care in 2 steps. First, potential quality indicators for each dimension of health care to be covered were defined. Different working groups were constituted each responsible for a specific healthcare domain. Within each working group clinicians, epidemiologists and experts in quality management were represented. Previously published recommendations and indicator sets as well as individual experience of the members of the working group were regarded for the preselection of indicators. Potential indicators were suggested by the working groups, reviewed and finally selected after consensus of the whole board.
In the second step, the selected indicators were evaluated according to the predefined methodological requirements of quality indicators, as well as published evidence in the literature and recommendations in guidelines or consensus statements. A systematic literature review was performed, and its results were presented in a standardized report for each indicator including rating on level of evidence. The rules of the Oxford Centre for Evidence-Based Medicine were applied for grading evidence.22 The board decided to implement an external review process by peers, as well as a prospective pilot study to further improve the quality of the project.18 Subsequently, the process of quality indicator development and its results was evaluated by independent external reviewers, who were not involved in the actual process. The reviewers were asked to comment on the appropriateness of used methods, practicability of defined indicators in routine care and coverage of all relevant aspects of acute in-hospital stroke care by the indicator set. In addition, a prospective pilot study was planned to investigate methodological properties of selected indicators and practicability of data collection in routine clinical care. Voluntary hospitals within the ADSR registers were invited to take part in the pilot study. Hospitals with different expertise in acute stroke care, defined by the number of patients treated per year, were included.
Selection of Quality Indicators
A detailed timetable of the whole process is presented in supplemental Table I (available online at http://stroke.ahajournals.org). In the first round, 34 potential indicators for measuring quality of care were defined by the board. A systematic literature review was performed investigating published evidence for each of these indicators. Overall, 15 national and international guidelines or consensus statements were analyzed and systematically scanned for recommendations for each indicator. The COCHRANE LIBRARY and PUBMED database were screened using specific keywords for each defined indicator. In total, 16 832 abstracts were identified and 252 publications were analyzed in detail. Considering available evidence from the literature, recommendations in guidelines and the predefined requirements for performance measures, all potential quality indicators were evaluated finally by the board. By consensus of the board, 9 of the 34 potential indicators were sorted out at present because they did not fulfill predefined requirements for performance measures or the evidence on their relevance was lacking (‘integration of a hospital in a regional network for admission of stroke patients’, ‘24-hour permanent attendance of a special stroke physician in hospital’, ‘availability of a specific physiotherapy and speech therapy department’, ‘integration of a hospital in a local treatment network with other hospitals’, ‘performance of a standardized patient assessment’, ‘preparation of an individual treatment- and rehabilitation-plan’, ‘realization of a treatment- or rehabilitation-plan’, ‘consultation of all patients by rehabilitation physician’, ‘performance of echocardiography in ischemic stroke patients’, ‘recurrent stroke during hospitalization’). In addition, 2 indicators were combined (‘internal quality management system’ and ‘participation on external audits’ into ‘implementation of an internal and external quality management system in the hospital’ and ‘availability of vascular imaging of carotid arteries’ and ‘availability of diagnostic cardiologic methods’ into ‘availability of diagnostic methods vascular imaging and diagnostic cardiologic methods at the hospital’), leaving in total 23 potential indicators.
For the purpose of the external review a detailed preliminary report was drafted for the independent experts, including an extended description of the methodological procedure and the results of the process with an extensive presentation of all indicators. Experts from the field of methodological development of quality indicators, acute stroke care, neurology, internal and geriatric medicine, as well as rehabilitation served as voluntary reviewers. Overall, the indicator set was judged to be appropriate by the reviewers; no relevant aspects of stroke care were missed. One indicator was recommended to be split in 2 indicators (‘early rehabilitation’ in ‘early rehabilitation-physiotherapy/occupational therapy’ and ‘early rehabilitation-speech therapy’), resulting now in total 24 potential indicators. Most remarks and comments of reviewers related to the presentation of indicators, especially to inclusion and exclusion criteria, case-mix adjustment and to description of specific definitions. As options in the future development of the indicator set, the experts suggested to also add indicators for prehospital processes as well as indicators adapted to the specific needs of inpatient rehabilitation in Germany. A final consensus decision about the indicator set was made by the board considering the recommendations of the reviewers. The whole indicator set was classified into 3 clusters with 7 indicators related to hospital structure, 14 indicators related to processes and outcome of patients during hospitalization and 3 indicators related to processes and outcome of patients at follow-up (supplemental Table II⇓, available online at http://stroke.ahajournals.org).
Prospective Pilot Study
A standardized questionnaire was developed that included the new indicator set and the variables needed to adjust for patient’s variability. In addition, a detailed manual of operation was prepared. Overall, 13 hospitals agreed to participate in the pilot study (92% departments of neurology; 85% provided Stroke Unit services; 31% treated <250, 31% 250 to 500 and 38% >500 stroke patients per year). All consecutively admitted patients with stroke or transient ischemic attack within these hospitals during a 2-month period were aimed to be documented. The expected number of patients treated within the hospitals during this time period was 1175. In total, data on 1006 patients with stroke or transient ischemic attack were collected yielding a documentation rate of 86% (median 96%, range per hospital 25% to 100%). A sufficient variance between hospitals in the performance of the indicators could be documented (Table). Overall the percentage fulfilled for several indicators was high. The largest variation occurred for the indicator ‘screening of patients for swallowing disorders’, the smallest for ‘brain imaging in stroke suspicious patients’. A standardized quality report for each indicator was prepared and distributed to the hospitals for comments (example in the Figure). The collection of the new indicator set was judged to be practicable by the physicians in the participating hospitals. The median time to document the required information for each patient was 5 minutes. Comments of participating hospitals related mainly to description of variables and presentation of results.
Implementation of Defined Quality Indicators in Germany
Based on the results of the pilot study, the questionnaire and the manual of operation was revised finally. The complete documentation for each quality indicator will be provided to interested readers on request by the study group. The executives of the participating ADSR registers aim to implement the new indicator set within the regional registers starting from April 2006. To adjust for changes in the German healthcare system or new trends in stroke treatment the board agreed on a periodical update of the quality indicators scheduled in 2-year time intervals.
This report summarizes the development and the implementation of evidence-based indicators for measuring quality of acute hospital stroke care by a multidisciplinary board in Germany. Over a 2-year-period, 24 quality indicators were developed and evaluated according to published evidence and predefined methodological requirements. The latter included a systematic literature review, an independent evaluation by external experts and a prospective pilot study on applicability. The indicator set was developed to measure the performance of an individual hospital in the healthcare dimensions structure, process and outcome and within the treatment phases prehospital, in-hospital/acute and postacute. The implementation of the quality indicators within regional stroke registers in Germany will start during 2006.
Quality improvement programs can actually improve acute stroke care in hospitals as recently demonstrated, eg, by increased rates of tissue plasminogen activator use23 or by lower pneumonia rates in hospitals with formal dysphagia screening protocols.24 One widely used tool to improve quality of hospital care is the regular feedback of the performance in specific indicators.15 Several initiatives to improve quality of stroke care in different countries have defined specific indicators to compare the performance of hospitals.9,11,13,14,12 Some similarities in the defined indicators between theses initiatives and our proposal can be found, especially regarding in-hospital processes. For example, all initiatives include indicators related to antithrombotics at discharge in eligible patients.9,11,13,14,12 The majority of the programs also include indicators related to Stroke Unit or multidisciplinary team care,11,14 to specific aspects of stroke education9,13,14,12 and to tissue plasminogen activator administration.9,13,14,12 Comparable to our results, some initiatives explicitly include indicators on organizational aspects11 or focus on long-term outcome of stroke patients to measure quality of care.14 However, the indicator sets used in the different quality improvement programs demonstrate also substantial variations. These differences can partly be explained by different methods in the development of the indicators and by different aims when developing performance measures. In addition, the characteristics of the specific healthcare system had to be considered because the experience from one country could not be completely transferred to other healthcare settings without adaptations to the specific local needs. For example, the performance measure ‘home visits before discharge’ from the Royal College of Physicians key indicators for stroke care11 could not be defined as an indicator for acute hospital care in Germany because of the strict separation between outpatient and inpatient care.
Our development process has strengths and limitations. We decided to develop quality indicators using a standardized, evidence-based approach following previously defined recommendations to ensure high methodological quality and maximal transparency of our results.8,18 A systematic literature review and an external evaluation were implemented as quality criteria to avoid the overemphasis of specific aspects of health care. The indicators were developed by a multidisciplinary board to guarantee a wide acceptance of the results by institutions and organizations engaged in acute stroke care in Germany. The whole process was implemented within the context of ongoing stroke registers in Germany to guarantee sustainability of the program and facilitate its implementation. The information required for calculating the defined indicators will be collected prospectively by the treating physicians. There is evidence that data feedback efforts must be perceived as valid by physicians to motivate changes.25 Thus, results of our quality initiative might be more able to change current practice compared with initiatives that use information from administrative records alone because the latter might be perceived as invalid. The annual pooled data analysis of the regional stroke registers within the ADSR will allow to evaluate prospectively the impact of the implementation of the indicator set on quality of acute stroke care in Germany. Our study has also weaknesses. Because the aim of the process was to develop indicators for in-hospital stroke care in Germany, no performance measures on a population level, eg, recurrence rates, could be considered. In addition, in Germany acute care hospitals have no direct influence in processes of prehospital emergency care. Thus, only prehospital indicators related to hospital structure, like training of emergency medical services, were included in the indicator set. Furthermore, the necessary resources for the data collection must further be minimized, eg, by facilitated electronic data capture. One major challenge to be solved is a regular follow-up of all stroke patients in Germany. Currently experience on indicators related to long-term follow-up of stroke patients in Germany is derived from different regional activities only.26 The broad follow-up of patients after discharge is actually hampered by logistics, costs and legal legislations in several federal states in Germany such as requirement of a written informed consent of the patient for a central follow-up. These problems also limit a regular follow-up of stroke patients in quality initiatives in other countries with similar legislation like Germany.27
The development of evidence-based indicators for measuring quality of acute stroke care in Germany is an important step toward a standardized audit of stroke care on a national level. The chosen standardized approach has ensured maximal transparency, acceptance and sustainability of the defined performance measures among all professions and organizations involved in acute stroke care in Germany.
We thank Dr Heidrich, Muenster, for his methodological input in developing the indicator set and Prof Haass, Homburg, and Prof Gottwick, Nuremberg, for their valuable contribution to the first two workshops. In addition, we want to express our honest gratitude to the voluntary reviewers of the development process: Prof R. Baumgartner, Department of Neurology, University of Zurich; Dr T. Rudd, Royal College of Physicians, London; Prof G. Stucki, Institute of Physical Medicine and Rehabilitation, University of Munich; Dr G. Blumenstock, Department of Medical Information Processing, University of Tuebingen.
Finally, we want to thank the following hospitals for their participation in the pilot phase:
Bad Zwesten: Neurologische Akutklinik; Bonn: Ev. Kliniken Bonn gGmbH Betriebsstätten Johanniter-Krankenhaus; Ebersberg: Abteilung für Innere Medizin der Kreisklinik; Emden: Neurologische Abteilung des Hans-Susemihl-Krankenhauses; Erlangen: Neurologische Klinik des Universitaetsklinikums; Hagen: Neurologische Klinik des St. Johannes Hospitals; Luedenscheid: Neurologische Klinik des Klinikums; Muenchen: Abteilung für Neurologie des Staedtischen Krankenhauses Harlaching; Muenster: Neurologische Klinik des Universitaetsklinikums; Nidda-Bad Salzhausen: Asklepios Neurologische Klinik Bad Salzhausen; Paderborn: Neurologische Klinik des St Vincenz Krankenhauses; Teupitz: Abteilung für Neurologie der Landesklinik; Westerstede: Neurologische Klinik der Ammerland Klinik
Sources of Funding
The study was partly supported by the German Federal Ministry of Research (BMBF) within the Competence Net Stroke. There was no financial or other support by other third parties for the working process or manuscript preparation.
This article was presented in part as an oral presentation at the Stroke 3-Country-Meeting in Vienna, 18 March 2006.
- Received June 15, 2006.
- Revision received July 13, 2006.
- Accepted July 21, 2006.
Sacco RL, Adams R, Albers G, Alberts MJ, Benavente O, Furie K, Goldstein LB, Gorelick P, Halperin J, Harbaugh R, Johnston SC, Katzan I, Kelly-Hayes M, Kenton EJ, Marks M, Schwamm LH, Tomsick T; American Heart Association; American Stroke Association Council on Stroke; Council on Cardiovascular Radiology Intervention; American Academy of Neurology. Guidelines for prevention of stroke in patients with ischemic stroke or transient ischemic attack: a statement for healthcare professionals from the American Heart Association/American Stroke Association Council on Stroke: cosponsored by the Council on Cardiovascular Radiology and Intervention: the American Academy of Neurology affirms the value of this guideline. Stroke. 2006; 37: 577–617.
Adams H, Adams R, Del Zoppo G, Goldstein LB; Stroke Council of the American Heart Association; American Stroke Association. Guidelines for the early management of patients with ischemic stroke: 2005 guidelines update a scientific statement from the Stroke Council of the American Heart Association/American Stroke Association. Stroke. 2005; 36: 916–923.
Hack W, Kaste M, Bogousslavsky J, Brainin M, Chamorro A, Lees K, Leys D, Kwiecinski H, Toni P, Langhorne P, Diener C, Hennerici M, Ferro J, Sivenius J, Gunnar N, Bath P, Olsen TS, Gugging M; European Stroke Initiative Executive Committee and the EUSI Writing Committee. European Stroke Initiative Recommendations for Stroke Management-update 2003. Cerebrovasc Dis. 2003; 16: 311–337.
Measuring and Improving Quality of Care: A Report From the American Heart Association/American College of Cardiology First Scientific Forum on Assessment of Healthcare Quality in Cardiovascular Disease and Stroke. Circulation. 2000; 101: 1483–1493.
Stroke Measure Set. Available at: http://www.jcaho.org/dscc/psc/stroke+measure+set.htm. Accessed on April 26, 2006.
Rudd AG, Pearson M. National stroke audit. Clin Med. 2002; 2: 496–498.
Kapral MK, Laupacis A, Phillips SJ, Silver FL, Hill MD, Fang J, Richards J, Tu JV; Investigators of the Registry of the Canadian Stroke Network. Stroke care delivery in institutions participating in the Registry of the Canadian Stroke Network. Stroke. 2004; 35: 1756–1762.
Reeves MJ, Arora S, Broderick JP, Frankel M, Heinrich JP, Hickenbottom S, Karp H, LaBresh KA, Malarcher A, Mensah G, Moomaw CJ, Schwamm L, Weiss P; Paul Coverdell Prototype Registries Writing Group. Acute stroke care in the US: results from 4 pilot prototypes of the Paul Coverdell National Acute Stroke Registry. Stroke. 2005; 36: 1232–1240.
Jamtvedt G, Young JM, Kristoffersen DT, Thomson O’Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2003; CD000259.
Holloway RG, Vickrey BG, Benesch C, Hinchey JA, Bieber J; National Expert Stroke Panel. Development of performance measures for acute ischemic stroke. Stroke. 2001; 32: 2058–2074.
http://www.cebm.net/levels_of_evidence.asp. Accessed on April 26, 2006.
Katzan IL, Hammer MD, Furlan AJ, Hixson ED, Nadzam DM; Cleveland Clinic Health System Stroke Quality Improvement Team. Quality improvement and tissue-type plasminogen activator for acute ischemic stroke: a Cleveland update. Stroke. 2003; 34: 799–800.
Hinchey JA, Shephard T, Furie K, Smith D, Wang D, Tonn S; Stroke Practice Improvement Network Investigators. Formal dysphagia screening protocols prevent pneumonia. Stroke. 2005; 36: 1972–1976.
Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Saf Health Care. 2004; 13: 26–31.
Audebert HJ, Kukla C, Clarmann von Claranau S, Kuhn J, Vatankhah B, Schenkel J, Ickenstein GW, Haberl RL, Horn M; TEMPiS Group. Telemedicine for safe and extended use of thrombolysis in stroke: the Telemedic Pilot Project for Integrative Stroke Care (TEMPiS) in Bavaria. Stroke. 2005; 36: 287–291.