Developing a Tool to Assess Quality of Stroke Care Across European Populations
The EROS Quality Assessment Tool
Background and Purpose—There are significant differences in the provision of care and outcome after stroke across countries. The European Registers of Stroke study aimed to develop, test, and refine a tool to assess quality of care.
Methods—We used a systematic review and grading of evidence for stroke care across the clinical pathway and developed and field-tested a quality tool that was delivered by post and later by site visit at 7 centers. Items were refined by using an algorithm that took into account the level of evidence, measurement properties, and consensus of opinion obtained using, the Delphi techniques.
Results—The tool included 251 items across 11 domains, of which 214 items could be categorized by any level of evidence. Overall agreement between postal and site visit modes of delivery was acceptable (κ=0.77), with most items having a κ >0.5. The refinement process resulted in 2 practical versions of the tool (93 items and 22 items). Positive responses to items in the tool indicated implementation of evidence-based stroke care. In field testing, the proportion of positive responses to evidence-based items ranged from 43% to 79% across populations. Proportions of different types of evidence being implemented were similar: high quality 62%, limited quality 72%, and expert opinion 54% across the populations. More than half (4 of 7) of the centers provided stroke unit care and thrombolysis, but availability and access to inpatient rehabilitation varied significantly, with poor access to community follow-up for rehabilitation and medical management.
Conclusions—The European Registers of Stroke Quality Assessment Tool has potential to be used as a framework to compare services and promote increased implementation of evidence-based care.
- stroke units
- stroke registers
- stroke management
- organized stroke care
- quality assessment
- performance measurement
European studies have shown that stroke service provision and outcomes vary considerably across Europe.1,–,4 Grieve and colleagues3 demonstrated in the late 1990s that there were significant variations in the costs of stroke care across Europe, with centers spending more per case generally having better outcomes, where the resource was deployed into stroke unit care. A more recent survey of European hospitals estimated that <10% that admitted acute stroke patients had optimal facilities, and in 40%, even the minimum level of facilities was unavailable.5
Increasingly, assessment of the performance of health services is required if health costs are to be contained. In many countries, national and European guidelines have been endorsed, yet the quality of, and methods used to develop these guidelines vary.6,–,8 To assess quality and implementation of evidence in clinical practice, in the early 2000s in the United States, a national multidisciplinary panel assisted in the development of 44 potential stroke performance measures and identified 21 that rated highly for acute care.9 More recently in the United States, performance measures have been developed for stroke and are being implemented by stroke quality improvement programs, although these measures are focused on process measures identified for acute hospital-based care.10
The European Registers of Stroke (EROS) study11,12 was a 4-year prospective, observational study of patients after their first episode of stroke. This study aimed to develop, test, and refine a tool to assess quality of care in 7 populations of Europe participating in the EROS study. Outcome data from 3- and 12-month follow-up in the EROS populations will be reported in a future article.
Developing the EROS Quality Assessment Tool
Synthesizing the Evidence of Effective Stroke Care
A comprehensive literature search of major electronic databases was undertaken to identify recommendations about key elements of the structure and process of stroke care (including Medline and the Cochrane Library) up to the start of the project (August 2003). The collaborating centers also provided details of their national and local guidelines and protocols (where these existed) that formed the basis of their stroke services. The evidence was categorized as 1=high quality, 2=limited evidence, 3=expert opinion, and 4=not supported by evidence6,7 (see Table 1).
Development of the Quality Assessment Tool
The key recommendations of each guideline were collated and allocated to different domains encompassing the whole patient pathway. Recommendations were rewritten as questions. The structure of the EROS Quality Assessment Tool (EQAT) was similar to that of the Royal College of Physicians' (London) audit tool13; this was supplemented with an additional 17 questions used in a previous study of stroke in Europe14 to gain a broader perspective of the non-UK healthcare systems. The EQAT contained 251 items grouped into 11 domains, as follows: specialist stroke services (45 items), management protocols (51 items), the multidisciplinary team (8 items), caregivers and family (25 items), acute specific medical diagnosis (11 items), acute medical and surgical interventions (9 items), early disability assessment and management (20 items), rehabilitation interventions (43 items), transfer back to the community (9 items), long-term management (14 items), and supplementary questions (various nonclinical topics; 16 items).
Only 214 of the 251 items could be categorized by any level of evidence (categories 1 to 3 in the preceding section and Table 1), with the majority of evidence from studies of acute hospital care; the rest (category 4) were questions about structures and processes in the various centers, for example, funding, access to voluntary support, and patient choice.
Testing the EQAT in the Centers
To examine feasibility, practicality, and reliability, the EQAT was used first as a postal questionnaire and second in a semistructured, face-to-face interview delivered at a site visit to 7 self-selected centers caring for stroke patients across Europe: Dijon (France), Florence (Italy), Stockholm (Sweden), Mahon (Menorca, Spain), Kaunas (Lithuania), Warsaw (Poland), and South East London (England).11 The postal version was sent to principal investigators in each of these centers and was returned several weeks before a site visit was conducted. These assessment visits to administer the EQAT were completed between October 2003 and August 2004 by an international, multidisciplinary group of up to 6 expert members who visited each center during a 2- to 3-day period. After all visits had been completed, the principal investigators were asked to consult with their multidisciplinary teams to evaluate each item in the EQAT by using the Delphi technique15 to ask the question: “Compared to the other items, how useful is this item in a measurement tool designed to measure quality of stroke care?”
Responses were made on a visual analog scale, anchored at one end with “least useful” and “most useful” at the other. Scores relating to “usefulness” were collated, anonymized, and sent back to the centers by the EROS center in Glasgow, thus allowing centers to see their own responses compared with anonymized responses from the other centers. A second round of the Delphi exercise was carried out, allowing centers to modify their responses after having seen the other scores.
Refining the Number of Items in the EQAT
Each of the original 251 items was scored on 3 aspects: first, the quality of the underlying evidence (1=low, 3=high); second, the measurement properties (reliability assessed by κ scores, where 3=high reliability); and third, the level of consensus on the utility of the item, based on the Delphi consensus data (3=high-level score for “usefulness”).
A simple algorithm, quality of evidence+measurement properties+ level of consensus=score/9, was used with equal weighting for each aspect to derive a score from a possible maximum of 9. Different configurations of the EQAT based on the rankings were produced for future field testing.
Descriptive statistics were used to analyze the results. κ Scores were calculated on the basis of agreement of responses between postal and site visit responses to the EQAT. Arbitrary cutoff points for agreement were designated as >0.79 for good agreement, 0.4 to 0.79 for moderate agreement, and <0.4 for poor agreement. For the Delphi consensus exercise, mean values from scores on the visual analog scale (scored 0 [minimum] to 100 [maximum] for usefulness) were calculated for each item, and these scores were included in the feedback for the second round. Mean scores from the second round were ranked by “usefulness” rating score per item. For the algorithm, total scores based on the 3 elements were calculated for each item and ranked by item for the full EQAT and within each domain.
Ethics approval was granted as part of the EROS project in each location.
Developing the EQAT
Two hundred fifty-one items were included in the EQAT; of these, 214 items were attributed to evidence at levels 1 to 3 (Table 1). Based on responses from the site visits, Table 2 shows the number of positive responses to those questions attributed with an evidence base in the 7 European centers, by level of evidence.
Overall agreement between responses to the EQAT when delivered by post versus site visit was considered acceptable (κ=0.77), with most items having a κ >0.5. Where there were discrepancies between the postal version and the site visit (taken as the “gold standard”), these generally featured overreporting, that is, reporting availability or provision of service features that were not corroborated by the visiting team of experts. This was observed in all centers to varying degrees. Much of the overreporting related to the existence of written information on policies and procedures that was not available for inspection during the visits.
Some individuals who responded to the postal questionnaire seemed not to have ready access to detailed information on specific professional roles (especially for nursing), processes, and interventions. There may also have been some lack of clear understanding by postal respondents for the terms “early discharge team” and “stroke-specific services,” as these were sometimes reported in the postal version but not found during the site visits. Similarly, there was some uncertainty in postal responses regarding the level of follow-up services routinely provided to patients, the use of clinical audit, and the availability of certain drugs. Feedback was provided to the centers from the first round of the Delphi exercise to rate the usefulness of items, allowing them to reconsider their scores against the mean score based on all centers' responses.
The scores remained largely stable from the first to the second round, with a 2.4% reduction in overall usefulness scores (range of changes, −7 to +9 points on the 100-point scale). Individual domains remained relatively stable. However, there was a positive shift toward the usefulness of questions in “early disability assessment and management” and “acute specific medical diagnosis.” There was a negative shift from the usefulness of questions in “additional service questions,” ”long-term care,” and “transfer back to the community,” many of which have a lower level of supporting evidence. Those items found to be more useful in the second round included questions on referral to speech and language therapists for those patients with an abnormal swallow, assessment of nutrition, screening for emotionalism, and access to multidisciplinary case notes. Those perceived to be less useful were questions concerning the number of pharmacy staff members, patients being excluded from insurance schemes on the basis of their condition, provision of food and laundry, availability of functional electric stimulation, the routine use of fluoxetine or amphetamine, availability of data on socioeconomic status, and which clinical outcome measures were used.
In the final round, EQAT items considered “most useful” (overall mean usefulness score >90 of a possible maximum of 100) were questions related to access to 24-hour medical care, review by senior medical specialist, access to imaging, 24-hour access to computed tomography scanning, computed tomography scanner on site, scanning before commencing anticoagulants, swallow test, access to speech and language therapist, and access to specialist stroke speech and language therapist.
Refinement of the EQAT According to the Algorithm
Selecting the highest-scoring items results in a tool that concentrates exclusively on acute care. However, if the domain structure of the original tool is maintained to select the highest-scoring items at different parts of the entire patient pathway, then a more balanced tool is obtained. Selecting items from the top 2 ranks in each domain resulted in a version with 93 items. Alternatively, by selecting the highest-scoring 2 items per domain (with the item with the highest “usefulness” score selected in cases of tied ranking), a 22-item version of the EQAT was produced.
Quality Assessed by the EQAT
Responses to the 93 items in the abbreviated version and full wording of the questions are given in supplemental Figure 1(http://stroke.ahajournals.org). Overall, all sites provide some form of specialist, primarily hospital-based, acute stroke services (supplemental Figure 1). Generally, good access to specialist stroke nursing and multidisciplinary care was reported. In contrast, community-based stroke care was less commonly reported. All sites that have existing acute stroke units had written management protocols in all key aspects of stroke management and secondary prevention, except rehabilitation. Limited patient-based audit was ongoing in the study centers.
Only 2 sites reported involving caregivers and family members in the provision of inpatient care. Levels of information for patients and caregivers displayed at the ward level varied among sites. The majority of the sites provided a variety of acute specific medical diagnostics, including imaging, emergency scanning, and chest x-ray. Thrombolysis services for relevant patients were available in 4 centers, and routine prophylactic anticoagulation was reported in 3 centers. Overall, all centers provided early disability assessment and management, in particular, a multidisciplinary team assessment within 5 days, assessment of the ability to swallow, appropriate management for patients unable to swallow, and nutritional support. However, the most effective intervention, provision of coordinated stroke unit care, remained limited, and proven methods of secondary prevention were not universally used.
This multinational study has demonstrated that it is feasible to develop an evidence-based tool to describe, in a standardized way, the quality of stroke care across the stroke care pathway in different European healthcare systems. The EQAT was able to identify variations among centers. Though still largely dominated by the evidence base for acute care, the EQAT also included community services and follow-up arrangements.
The items in the EQAT were derived from evidence-based guidelines and rated only according to the quality of their supporting evidence. A recent study from Sweden16 used novel, transparent methods for prioritizing items on guidelines, based not just on the level of scientific evidence but also on perceived need, effectiveness of action, cost-effectiveness, and involving broad consultation with stakeholders. Although similar methods could be applied to prioritize items on a tool to assess quality of care, the social and cultural context may be specific to each country, thereby limiting the generalizability of that approach.
A recent postal survey of facilities available in a large, random sample of European hospitals treating stroke5 demonstrated the very limited provision of facilities for the majority of acute stroke patients in Europe. However, this survey focused on acute care facilities only; chronic diseases such as stroke are typified by the requirement of care delivery across service boundaries; prolonged, complex episodes of care; or different care requirements at various stages of the disease. We have attempted to gather data from a wide spectrum of care, and our questionnaire included items relevant to most stages of stroke management. National audits may demonstrate a breadth of coverage of stroke service issues but lack the depth of this approach. Such methods could be used to accredit stroke units or stroke services in the drive to improve uptake of evidence-based practice.17
The EQAT allowed the systematic identification of examples of good clinical care, with many similarities11across centers. However, despite the availability of high-quality evidence, structures and processes are still suboptimal, that is, less than ≈70% compliance where evidence was available. Reasons for differences in the uptake and implementation of available evidence are worthy of further study. For example, the 2 centers where quality was notably limited are from diverse healthcare systems and are very different in character: one, a small service treating few patients, the other a large service treating many but with limited resources. The factors influencing quality service provision are likely to be different in each. The availability of the most effective intervention (coordinated stroke unit care18) remains limited, reflecting results from recent large-scale national19,20 and international5,21 studies.
The EQAT would benefit from further development to examine its validity and method of delivery. With 251 items, the questionnaire is feasible with dedicated input, but we aim to modify and condense it to be more practical in routine application. We included the evidence-based guidelines available at the start of the project. However, evidence continues to evolve and become incorporated into contemporary guidelines, for example, the European Stroke Organisation guidelines in 2009.7 Although at present the EQAT does not appear to be missing many key items, developments in stroke care with a strong evidence base would need to be included in any future revision.
Because the centers were self-selecting, it is not possible to generalize results from our opportunistic sample across regions. The EROS registers will gather information to allow for case-mix adjustment, but to establish whether the services are typical, the questionnaire would need to be administered to other centers in the collaborating countries. However, the project did provide the opportunity to examine areas where we expected a diversity of care. It is possible that responses to the questionnaire were subject to response bias, either overestimating or underestimating the quality of care. The site visits, with several interviewers given the opportunity to ask questions and triangulate responses along with limited direct observational studies,22 may have addressed this to some extent.
Our study is still somewhat focused toward inpatient management, partly reflecting the ease with which we could validate responses, for example, on the site visits. Care tended to be concentrated in the larger acute centers and was less easily witnessed in the community. In its reduced form, the EQAT requires further validation and testing in different settings. Some may question the arbitrary nature of the weightings used in the algorithm and the cutoff values chosen. However, the shorter version of the EQAT appears to have simple face validity and may be more practical than the longer version.
We have developed a questionnaire schedule and used it on site visits to describe stroke services in 7 European centers. Substantial variation exists in service provision. Further validation of this approach is required to test the reliability of the questionnaire and to link it to patient outcomes, but it forms an evidence-based tool with which to assess quality of stroke care. We plan to condense the EQAT in the future to make it more practical.
Sources of Funding
The study was funded by the European Commission, Fifth Framework Programme. C.D.A.W., A.G.R., and I.W. acknowledge financial support from the UK Department of Health via the National Institute for Health Research Comprehensive Biomedical Research Centre award to Guy's & St Thomas' NHS Foundation Trust in partnership with King's College London and King's College Hospital NHS Foundation Trust, London, England. C.M. is funded by a UK Department of Health Career Scientist award.
The cooperation of all staff members in the centers who assisted with the site visits and I. Sayed, study coordinator, is acknowledged.
The following institutions and persons participated in the EROS project.
Writing Group: Division of Health and Social Care Research, King's College, London, England: Ian Wellwood, PhD; Christopher McKevitt, PhD; Charles D.A. Wolfe, FFPH; National Institute for Health Research Comprehensive Biomedical Research Centre at Guy's & St Thomas' NHS Foundation Trust, in partnership with King's College London and King's College Hospital NHS Foundation Trust, London, England: UK Ian Wellwood, PhD; Charles D.A. Wolfe, FFPH; University of Glasgow, Glasgow Scotland: Section of Public Health and Health Policy, Olivia Wu, PhD, and Academic Section of Geriatric Medicine, Peter Langhorne, PhD; Institute of Neurosciences, Italian National Research Council, and Department of Neurological and Psychiatric Sciences, University of Florence, Florence, Italy: Antonio Di Carlo, MD; Guy's & St Thomas' NHS Foundation Trust, London, England: Anthony G. Rudd, FRCP; University of Burgundy, University Hospital of Dijon, Dijon, France: Yannick Bejot, MD; First Department of Neurology, Institute of Psychiatry and Neurology, Warsaw, Poland: Danuta Ryglewicz, MD, PhD; Institute of Cardiology, Kaunas University of Medicine, Kaunas, Lithuania: Daiva Rastenyte, MD.
Coinvestigators: Dijon: M. Giroud, MD (Stroke Registry of Dijon, University of Burgundy, University Hospital of Dijon; local Principal Investigator [PI]); Sesto Fiorentino: D. Inzitari, MD (Department of Neurological and Psychiatric Sciences, University of Florence; Institute of Neurosciences, Italian National Research Council, Florence; local PI); London: C. McKevitt, PhD (Division of Health and Social Care Research, King's College London; local PI); Menorca: M. Torrent, MD (Area de Salud de Menorca, ib-salut, Menorca; local PI); Warsaw: H. Sienkiewicz-Jarosz, MD, PhD (First Department of Neurology, Institute of Psychiatry and Neurology, Warsaw; local PI); A. Czlonkowska, PhD (Second Department of Neurology, Institute of Psychiatry and Neurology, Warsaw; local PI). Data Centre: C. Sarti, PhD (National Public Health Institute, Helsinki, Finland; local PI); Martin Dennis, MD (University of Edinburgh; local PI).
Contributors: Sesto Fiorentino: M. Lamassa, MD (collection of data), P. Nencini, MD (collection of data), A. Poggesi, MD (collection of data), F. Pescini, MD (collection of data), A. Cramaro, MD (collection of data), E. Magnani, MD (collection of data), Department of Neurological and Psychiatric Sciences, University of Florence; M. Baldereschi, MD (Institute of Neurosciences, Italian National Research Council, Florence, collection of data); Kaunas: D. Sopagiene, MD (collection of data), D. Kranciukaite, MD (collection of data), Institute of Cardiology, Kaunas University of Medicine, Kaunas; London: I. Sayed, MA (Division of Health and Social Care Research, King's College London, collection of data); Menorca: J. Rodriguez-Mera, MD (Area de Salud de Menorca, ib-salut, Menorca, collection of data); Warsaw: M. Głuszkiewicz, MD (Second Department of Neurology, Institute of Psychiatry and Neurology, Warsaw, collection of data), J. Pniewski, MD (Neurology Department, Medical Research Centre, Polish Academy of Sciences/CSK MSWiA, Warsaw, collection of data); Data Centre: V. Moltchanov, PhD (National Public Health Institute, Helsinki, Finland, technical assistance).
The online-only Data Supplement is available at http://stroke.ahajournals.org/content/full/stroke.110.598938/DC1.
Natan M. Bornstein, MD, was the Consulting Editor for this paper.
- Received August 10, 2010.
- Accepted January 13, 2011.
- © 2011 American Heart Association, Inc.
- Beech R,
- Ratcliffe M,
- Tilling K,
- Wolfe C
- Wolfe C,
- Tilling K,
- Beech R,
- Rudd A
- Grieve R,
- Hutton J,
- Bhalla A,
- Rastenytë D,
- Ryglewicz D,
- Sarti C,
- Lamassa M,
- Giroud M,
- Dundas R,
- Wolfe CD
- Clarke A,
- Rao M
- Leys D,
- Ringelstein B,
- Kaste M,
- Hacke W
Royal College of Physicians Intercollegiate Stroke Working Party. National Clinical Guidelines for Stroke, 3rd ed. London: Royal College of Physicians; 2008.
EUSI, European Stroke Organisation. Guidelines for stroke management, updated January 2009. http://www.eso-stroke.org/recommendations.php. Last accessed July 2010.
American Heart Association. Definitive stroke guidelines collection. http://www.americanheart.org/presenter.jhtml?identifier=3004586. Last accessed July 2010.
- Holloway R,
- Vickrey B,
- Benesch C,
- Hinchey J,
- Bieber J
- Reeves M,
- Parker C,
- Fonarow G,
- Smith E,
- Schwamm L
European Registers of Stroke (EROS) Collaborative Group. European Registers of Stroke. http://phs.kcl.ac.uk/eros/. Last Accessed June 1, 2009.
The European Registers of Stroke (EROS) Investigators. Incidence of stroke in Europe at the beginning of the 21st century. Stroke. 2009;40:1557–1563.
Royal College of Physicians Intercollegiate Working Party for Stroke. National Clinical Guidelines for Stroke, 2nd ed. London: Royal College of Physicians; 2004.
- McKevitt C,
- Beech R,
- Pound P,
- Rudd A,
- Wolfe C
- Jones J,
- Hunter D
- Norving B,
- Wester P,
- Sunnerhagen K,
- Terént A,
- Sohlberg A,
- Berggren F,
- Wester P,
- Asplund K
- Rudd A,
- Hoffman A,
- Irwin P,
- Pearson M,
- Lowe D
Stroke Unit Trialists' Collaboration. Organised inpatient (stroke unit) care for stroke. Cochrane Database Syst Rev. 2006;2:CD000197.
- I. Wellwood I,
- Langhorne P,
- McKevitt C,
- Bernhardt J,
- Rudd A,
- Wolfe C