This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Wilson, J.T. L. Articles by Bone, I. Search for Related Content PubMed PubMed Citation Articles by Wilson, J.T. L. Articles by Bone, I. Related Collections Health policy and outcome research Behavioral/psychosocial - stroke Behavioral Changes and Stroke

(Stroke. 2002;33:2243.)
© 2002 American Heart Association, Inc.

Original Contributions

Improving the Assessment of Outcomes in Stroke

Use of a Structured Interview to Assign Grades on the Modified Rankin Scale

J.T. Lindsay Wilson, PhD; Asha Hareendran, PhD; Marie Grant, BSc; Tracey Baird, MD; Ursula G.R. Schulz, MD; Keith W. Muir, MD Ian Bone, MD

From the Department of Psychology, University of Stirling, Stirling, UK (J.T.L.W., M.G.); Outcomes Research, Pfizer Ltd, Sandwich, UK (A.H.); and Department of Neurology, Institute of Neurological Sciences, Glasgow, UK (T.B., U.G.R.S., K.W.M., I.B.).

Correspondence to J.T.L. Wilson, Department of Psychology, University of Stirling, Stirling FK9 4LA, UK. E-mail J.T.L.Wilson{at}stir.ac.uk

	Abstract

Top Abstract Introduction Methods Results Discussion References

 
Background and Purpose— The modified Rankin Scale is widely used to assess changes in activity and lifestyle after stroke, but it has been criticized for its subjectivity. The purpose of the present study was to compare conventional assessment on the modified Rankin Scale with assessment through a structured interview.

Methods— Sixty-three patients with stroke 6 to 24 months previously were interviewed and graded independently on the modified Rankin Scale by 2 observers. These observers then underwent training in use of a structured interview for the scale that covered 5 areas of everyday function. Eight weeks after the first assessment, the same observers reassessed 58 of these patients using the structured interview.

Results— Interrater reliability was measured with the statistic (weighted with quadratic weights). For the scale applied conventionally, overall agreement between the 2 raters was 57% (_w=0.78); 1 rater assigned significantly lower grades than the other (P=0.048). On the structured interview, the overall agreement between raters was 78% (_w=0.93), and there was no overall difference between raters in grades assigned (P=0.17). Rankin grades from the conventional assessment and the structured interview were highly correlated, but there was significantly less disagreement between raters when the structured interview was used (P=0.004).

Conclusions— Variability and bias between raters in assigning patients to Rankin grades may be reduced by use of a structured interview. Use of a structured interview for the scale could potentially improve the quality of results from clinical studies in stroke.

Key Words: clinical trials • disability evaluation • outcome • outcome assessment

	Introduction

Top Abstract Introduction Methods Results Discussion References

 
The Rankin Scale¹ has wide acceptance as a measure of functional outcome after stroke and has become one of the most popular end points for clinical trials.² The version of the scale most commonly used in trials is the modified Rankin Scale,³ which is a simple 6-point assessment that includes reference to both limitations in activity and changes in lifestyle.

The reliability of the modified Rankin Scale has been investigated and found to be satisfactory.³ However, comparison with the Barthel Index⁴ indicates lower levels of interrater agreement for the modified Rankin Scale⁵ and suggests that some raters may systematically assign higher or lower Rankin grades than others. The descriptions given for the categories of the modified Rankin Scale are broad and open to subjective interpretation. Walking is the only explicit criterion for assessment mentioned, and even for this criterion, it is not specified whether someone requiring an aid should be considered able to walk. It is therefore left open to raters to develop idiosyncratic criteria or to apply the scale in an impressionistic manner. Discrepancies are particularly striking for Rankin grades 2, 3, and 4, and it has been suggested that interviewers should use a checklist of activities of daily living (ADL) to produce greater uniformity in the application of the scale.³ Wolfe and colleagues⁵ have advocated using Barthel scores to generate ratings on the Rankin Scale and shown that this improves reliability. However, this approach can be applied only to the lower outcome categories of the modified Rankin Scale that relate to the basic ADL assessed by the Barthel Index.

The Glasgow Outcome Scale⁶ is similar in concept to the Rankin Scale, and the problem of impressionistic use of the Glasgow Outcome Scale has been addressed through the use of a structured interview.⁷ The purpose of the present study was to develop a structured interview for the modified Rankin Scale and to compare this form of assessment with the conventional application of the scale. We wanted to investigate whether using a structured interview could increase agreement between raters on the modified Rankin Scale. We carried out 2 interrater reliability studies with the same patients 8 weeks apart, the first with the conventional scale and the second with the structured interview. To reduce the possibility of change between the first and second assessments, only patients who had suffered stroke 6 months previously were included in the study.

	Methods

Top Abstract Introduction Methods Results Discussion References

 
Structured Interview
The structured interview (Table 1) differs from the conventional guided interview for the modified Rankin Scale by defining specific questions to grade each category. The structured interview developed for the study consists of 5 sections: (1) constant care, (2) basic ADL, (3) instrumental ADL, (4) limitations in participation in usual social roles, and (5) checklist for the presence of common stroke symptoms. Items for inclusion in the interview were selected after review of outcome assessments used in stroke and focus groups held with stroke patients. An initial draft of the interview was piloted before the final version for the study was produced. Section 2 of the interview is based on items from the Barthel Index and followed the definitions provided by Collin et al.⁸ Sections 3 and 4 are adapted from the structured interview for the extended Glasgow Outcome Scale.⁷ Unlike a questionnaire, a structured interview allows reformulation of questions to suit the particular circumstances. In each section, restrictions in activities before stroke are recorded, and raters are instructed to discount preexisting limitations in the final rating. Raters were encouraged to interview a relative when possible and to base their assessments on ability to do the task rather than performance. Further details concerning the principles involved in administering a structured interview are given elsewhere.⁷The structured interview took 15 minutes to administer.

View this table: [in this window] [in a new window]   Table 1. The Modified Rankin Scale and Corresponding Sections of the Structured Interview

Participants
The study was confined to patients surviving stroke by 6 months. Study inclusion criteria were as follows: age 18 years; diagnosis of stroke 6 to 24 months previously; living at home, living in an institution, and/or attending outpatient clinics; and ability to respond appropriately to interview in English. Excluded from the study were patients with terminal cancer, seizure disorder, dementia, substance or alcohol abuse, and major organ failure (unstable cardiopulmonary function, impaired hepatic or renal function resulting in episodic alterations in functional ADL); those unable and/or unlikely to comprehend and follow the study protocol; and patients not contacted on the advice of their general practitioners. Informed consent was obtained for each study participant.

Procedure
Both raters were neurologists in training; rater 1 (T.B.) was a specialist registrar in neurology with 4 years of experience; and rater 2 (U.G.R.S.) was a senior house officer with 2 years of experience. Before beginning the study, the raters practiced applying the modified Rankin Scale in a stroke population. Patients were assessed on 2 occasions 8 weeks apart. On the first occasion, the 2 raters interviewed each patient independently and assigned a rating on the modified Rankin Scale. The raters were instructed not to confer about ratings of individual patients. Rankin grades were assigned immediately after the initial interview. After all patients had been assessed, the raters were trained to use the structured interview to assign Rankin grades. The patients were then recalled, and each patient was independently assessed with the structured interview.

Statistical Analysis
Strength of agreement between raters is described with the statistic that corrects for agreement by chance. When there are >2 points on an assessment, it is appropriate to use a weighted value (_w) to take into account the size of disagreements. To facilitate comparison with previous studies, we used quadratic weights for this analysis. Quadratic weights penalize extreme disagreements particularly heavily (differences are squared), and it has been shown that when weighted this way, _w is comparable to the intraclass correlation coefficient used for continuous measures.⁹ Brennan and Silman¹⁰ suggest the following interpretation of the statistic (weighted appropriately) for the agreement between clinical measures: 0 to 0.20=poor, 0.21 to 0.40=fair, 0.41 to 0.60=moderate, 0.61 to 0.80=good, and 0.81 to 1.00=very good. The 95% CIs for _w values are given. Ratings were also compared through the use of appropriate nonparametric tests.

	Results

Top Abstract Introduction Methods Results Discussion References

 
Sixty-three patients were initially recruited into the study and took part in the first assessment in which the modified Rankin Scale was applied in a conventional manner; 58 patients returned (92%) for a second assessment 8 weeks after the first with the Rankin structured interview. Reasons for loss to follow-up were death (n=1), serious illness (n=2), alcohol abuse (n=1), and did not attend (n=1).

The 58 patients (31 men) who took part in both assessments were between 37 and 90 years of age (mean, 68.3 years; SD, 10.95 years). The first assessment took place 6 to 24 months after stroke (mean, 17.1 months; SD, 5.2 months).

Ratings from the first interview, in which the modified Rankin Scale was applied in the conventional manner, are given in Table 2. The overall agreement between raters was 57%; the unweighted statistic was 0.44, and _w was 0.78 (95% CI, 0.53 to 1.0). In 8 cases, rater 1 rated patients less favorably than rater 2, and in 17 cases, rater 1 rated patients more favorably. Comparison of grades given by raters indicated a significant overall difference between observers (Wilcoxon Z=-1.98, P=0.048, 2-tailed test).

View this table: [in this window] [in a new window]   Table 2. Modified Rankin Scale: Cross-Tabulation of Rankin Grades Assigned by 2 Raters Who Independently Interviewed 58 Patients

Ratings from the structured interview for the modified Rankin Scale are given in Table 3. The overall agreement between raters was 78%, =0.70, and _w=0.93 (95% CI, 0.67 to 1.0). In 4 cases, patients were rated less favorably by rater 1, and in 9 cases, they were rated more favorably. There was no significant difference in the overall rankings assigned (Wilcoxon Z=-1.4, P=0.17).

View this table: [in this window] [in a new window]   Table 3. Structured Interview: Cross-Tabulation of Rankin Grades Assigned by 2 Raters for 58 Patients

To compare the studies, we analyzed disagreements between raters. There were 25 disagreements between raters in the first study and 13 in the second study. Rankin grades for rater 2 were subtracted from Rankin grades for rater 1, and the absolute differences with and without the structured interview were compared. The analysis showed that the extent of disagreement was less when the structured interview was used (Wilcoxon Z=-2.85, P=0.004).

The overall distributions of ratings for each rater (given in the "total" columns in Tables 2 and 3) do not differ substantially between the 2 assessments. Although there were differences in the individual ratings between the 2 assessments (22 for rater 1, 19 for rater 2), only 1 difference was by >1 category. To test whether there was a significant shift in overall scoring for each rater, we compared the Rankin scores given without the structured interview with those obtained with the structured interview using the Wilcoxon test. For both observers, there was no overall difference in the ratings assigned on the 2 assessments (Wilcoxon Z=-1.8, P=0.072 for rater 1; Z=-0.69, P=0.491 for rater 2). The 2 assessments were also highly correlated (Spearman’s correlation, 0.82, P<0.001 for rater 1; Spearman’s correlation, 0.90; P<0.001 for rater 2).

	Discussion

Top Abstract Introduction Methods Results Discussion References

 
The results for the conventionally applied modified Rankin Scale indicate good interrater reliability but indicate that significant bias may be present. The structured interview for the modified Rankin Scale had very good interrater reliability, the extent of disagreement between raters was less, and significant bias was not present. Comparison of ratings on the structured interview and the conventional Rankin showed that they were highly correlated, indicating that the structured interview has satisfactory criterion validity when measured against the conventional Rankin as a standard.

The findings of the present study are consistent with previous studies of the reliability of the modified Rankin Scale.^3,5,11 The interrater reliability of conventional assessment with the modified Rankin Scale is satisfactory but nonetheless open to improvement. Direct comparison of the present findings with previous reports is complicated because the distributions of gradings differ. The recruitment criteria for the present study tended to eliminate the most mildly disabled and the most severely disabled groups. To allow comparison with the present study, we reanalyzed the study of Van Swieten et al,³ confining analysis to 67 patients in Rankin categories 0 to 4. This analysis yielded an overall agreement of 61%, a of 0.49 and a _w of 0.80 (quadratic weights). Wolfe et al⁵ reported values of _w ranging from 0.75 to 0.96 for interobserver agreement on the Modified Rankin, and Bamford et al¹¹ gave a value of 0.72 for the version of the Rankin Scale used in the Oxfordshire Community Stroke Project. The reliability of the conventional Rankin Scale in the present study is thus very similar to previous reports. In agreement with the study of Wolfe et al,⁵ the present study also demonstrates that significant bias may be present even when the value is satisfactory. Wolfe et al reported systematic differences in the overall rankings produced by their 3 raters.

Limitations of the present study are that only 2 raters were used, and both came from similar professional backgrounds. In large clinical trials, multiple observers contribute data, and 2 observers do not represent this situation. Using multiple observers with different backgrounds may lead to greater divergence in the application of the conventional Modified Rankin (ie, lower reliability), and there could consequently be a larger effect of introducing a standardized procedure. In the present study, it was not possible to counterbalance order of assessment with the conventional and structured interview, because exposure to the structured interview will inevitably affect the style of approach adopted in the conventional assessment. Once exposed to the structured interview, raters may simply continue to ask the same questions when asked to assess patients on the conventional scale. It is possible that some of the reduction in variability is due to a practice effect for raters or another time difference. However, interrater reliability for the conventional assessment obtained in our study was similar to previous reports; and interrater reliability with the structured interview was similar to that found for the use of a structured interview for the Glasgow Outcome Scale in head-injured patients.⁷ Our results are therefore comparable with previous studies and suggest that major differences resulting from practice are unlikely. Further study could define the relevance of any time effects and investigate interrater reliability when the assessment is used by multiple raters from different professional backgrounds. After the study, the raters were debriefed in detail, and reasons for disagreements were identified. We have subsequently developed a set of detailed guidelines for the interview and a video for use in training raters. Training raters is particularly likely to be of importance in multicenter clinical trials, and raters should have the opportunity to observe an interview being conducted and record responses themselves. It may also be desirable to have an accreditation process to ensure that observers are applying the scale appropriately.

Van Swieten et al³ noted that further improvement in the Rankin Scale should be possible, and our results confirm that it is possible to reduce variability in ratings. The present findings probably have most relevance to the conduct of multicenter clinical trials. Use of the modified Rankin Scale as a functional assessment in clinical trials is supported by a recent analysis,¹² and it is likely to remain a popular end point. Demonstrating convincing treatment effects in acute stroke trials has proved to be a challenge, and it has been suggested that use of less-than-optimal methods to measure outcome may be responsible for problems of inconsistent findings.² Choi and colleagues¹³ have demonstrated that misclassification not only reduces the power of a clinical trial but also reduces the size of the observed treatment effect on dichotomous outcomes. The present investigation shows that use of a structured interview for the modified Rankin Scale may help to reduce variation between raters and improve the quality of results in clinical studies.

	Acknowledgments

 
This project was supported by a research grant from Pfizer UK to the University of Stirling. We would like to thank Peter J. Snyder, PhD, Robert Bagdorf, MD, and Michael Krams, MD, for their comments and support of the project.

	Footnotes

 
Copies of the Structured Interview for the Modified Rankin Scale and accompanying notes can be obtained from the corresponding author or at http://www.stir.ac.uk/psychology/staff/JTLW1.

Received October 31, 2001; revision received May 6, 2002; accepted May 10, 2002.

	References

Top Abstract Introduction Methods Results Discussion References

 

1. Rankin J. Cerebral vascular accidents in patients over the age of 60: II. Prognosis Scottish Med J. 1957; 2: 200–215.
2. Duncan PW, Jorgensen HS, Wade DT. Outcome measures in acute stroke trials: a systematic review and some recommendations to improve practice. Stroke. 2000; 31: 1429–1438.[Abstract/Free Full Text]
3. van Swieten JC, Koudstaal PJ, Visser MC, Schouten HJ, van Gijn J. Interobserver agreement for the assessment of handicap in stroke patients. Stroke. 1988; 19: 604–607.[Abstract/Free Full Text]
4. Mahoney FI, Barthel DW. Functional evaluation: the Barthel index. Maryland State Med J. 1965; 14: 61–65.
5. Wolfe CDA, Taub NA, Woodrow EJ, Burney PG. Assessment of scales of disability and handicap for stroke patients. Stroke. 1991; 22: 1242–1244.[Abstract/Free Full Text]
6. Jennett B, Bond M. Assessment of outcome after severe brain damage: a practical scale. Lancet. 1975; 1: 480–484.[Medline] [Order article via Infotrieve]
7. Wilson JTL, Pettigrew LEL, Teasdale GM. Structured interviews for the Glasgow Outcome Scale and Extended Glasgow Outcome Scale: guidelines for their use. J Neurotrauma. 1998; 15: 573–585.[Medline] [Order article via Infotrieve]
8. Collin C, Wade DT, Davies S, Horne V. The Barthel ADL Index: a reliability study. Int Disabil Stud. 1988; 10: 61–63.[Medline] [Order article via Infotrieve]
9. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation as measures of reliability. Educ Psychol Meas. 1973; 33: 613–619.[CrossRef]
10. Brennan P, Silman A. Statistical methods for assessing observer variability in clinical measures. BMJ. 1992; 304: 1491–1494.[Medline] [Order article via Infotrieve]
11. Bamford JM, Sandercock PAG, Warlow CP, Slattery J. Interobserver agreement for the assessment of handicap in stroke patients. Stroke. 1989; 20: 828.Letter.[Medline] [Order article via Infotrieve]
12. Broderick JP, Lu M, Kothari R, Levine SR, Lyden PD, Haley EC, Brott TG, Grotta J, Tilley BC, Marler JR, Frankel M. Finding the most powerful measures of the effectiveness of tissue plasminogen activator in the NINDS tPA Stroke Trial. Stroke. 2000; 31: 2335–2341.[Abstract/Free Full Text]
13. Choi SC, Clifton GL, Marmarou A. Misclassification and treatment effect on primary outcome measures in clinical trials of severe neurotrauma. J Neurotrauma. 2002; 19: 17–22.[CrossRef][Medline] [Order article via Infotrieve]

This article has been cited by other articles:

				J. H. Pan, X. Y. Song, S. Y. Lee, and T. Kwok Longitudinal Analysis of Quality of Life for Stroke Survivors Using Latent Curve Models Stroke, October 1, 2008; 39(10): 2795 - 2802. [Abstract] [Full Text] [PDF]

				R. S.K. Lo, J. O.Y. Cheng, E. M.C. Wong, W. K. Tang, L. K.S. Wong, J. Woo, and T. Kwok Handicap and Its Determinants of Change in Stroke Survivors: One-Year Follow-Up Study Stroke, January 1, 2008; 39(1): 148 - 153. [Abstract] [Full Text] [PDF]

				T. J. Quinn, K. R. Lees, H.-G. Hardemark, J. Dawson, and M. R. Walters Initial Experience of a Digital Training Resource for Modified Rankin Scale Assessment in Clinical Trials Stroke, August 1, 2007; 38(8): 2257 - 2261. [Abstract] [Full Text] [PDF]

				J. L. Banks and C. A. Marotta Outcomes Validity and Reliability of the Modified Rankin Scale: Implications for Stroke Clinical Trials: A Literature Review and Synthesis Stroke, March 1, 2007; 38(3): 1091 - 1096. [Abstract] [Full Text] [PDF]

				J. T. L. Wilson, A. Hareendran, A. Hendry, J. Potter, I. Bone, and K. W. Muir Reliability of the Modified Rankin Scale Across Multiple Raters: Benefits of a Structured Interview Stroke, April 1, 2005; 36(4): 777 - 781. [Abstract] [Full Text] [PDF]

				S. M. Greenberg, J. A. Eng, M. Ning, E. E. Smith, and J. Rosand Hemorrhage Burden Predicts Recurrent Intracerebral Hemorrhage After Lobar Hemorrhage Stroke, June 1, 2004; 35(6): 1415 - 1420. [Abstract] [Full Text] [PDF]

				F. B. Young, K. R. Lees, and C. J. Weir Strengthening Acute Stroke Trials Through Optimal Use of Disability End Points Stroke, November 1, 2003; 34(11): 2676 - 2680. [Abstract] [Full Text] [PDF]

				N. J. Newcommon, T. L. Green, E. Haley, T. Cooke, M. D. Hill, J.T.L. Wilson, A. Hareendran, T. Baird, K.W. Muir, and I. Bone Improving the Assessment of Outcomes in Stroke: Use of a Structured Interview to Assign Grades on the Modified Rankin Scale Stroke, February 1, 2003; 34(2): 377 - 378. [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Request Permissions Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Wilson, J.T. L. Articles by Bone, I. Search for Related Content PubMed PubMed Citation Articles by Wilson, J.T. L. Articles by Bone, I. Related Collections Health policy and outcome research Behavioral/psychosocial - stroke Behavioral Changes and Stroke