Simplified Modified Rankin Scale Questionnaire
Reproducibility Over the Telephone and Validation With Quality of Life
Background and Purpose—The simplified modified Rankin Scale questionnaire (smRSq) enables a reliable and rapid determination of the modified Rankin Scale score after stroke. We test the reliability and validity of a slightly revised smRSq.
Methods—Fifty consecutive outpatients 4.83±3.00 months after stroke were scored with a slightly revised smRSq by 3 raters selected consecutively from a list of 10: 4 stroke faculty, 3 neurology residents, 2 medial students, and 1 stroke research coordinator. Two ratings were in person within 20 minutes of each other and 1 was by telephone 1 to 3 days later. The telephone rating also included a quality of life scale, the Short-Form-12v2. Each rater was blinded to the other raters' scores.
Results—The average estimated time to administer the smRSq was 1.29 minutes (range, 0.50 to 2.25 minutes). The in-person raters agreed 78% (κ=0.71; CI, 0.57 to 0.86 and weighted κ [κw]=0.86; CI, 0.79 to 0.94). The first in-person and telephone raters agreed 82% (κ=0.76; CI, 0.63 to 0.90 and κw=0.87; CI, 0.79 to 0.95). The second in-person and telephone rates agreed 82% (κ=0.77; CI, 0.63 to 0.90 and κw=0.89; CI, 0.82 to 0.96). The smRSq correlated with the physical (r=−0.50, P=0.005) than the mental (r=−0.36, P=0.048) components of the Short-Form-12v2.
Conclusions—The slightly revised smRSq appears to be useful in clinical stroke; it has excellent reliability in person and by telephone, can usually be administered in <1.5 minutes by a wide variety of raters, and correlates with quality of life.
Reliability (interrater agreement) and feasibility, including telephone assessment, of valid outcome measures in stroke are essential for reproducible and efficient determination of outcomes. The modified Rankin Scale (mRS)1 remains a popular validated functional outcome measure in acute stroke trials and other stroke studies.2 However, its usefulness has been limited by suboptimal reliability,3 and there are limited data from complete reports about its reliability over the telephone.4,5 We recently set out to improve the reliability of the mRS at the same time as keeping the assessments relatively simple and short and preserving the construct and validity of the original mRS. We created 5 relatively simple questions that address the key functional states assigned to each mRS score, the simplified mRS questionnaire (smRSq).6 This questionnaire requires yes or no answers by a patient or a caregiver, takes on average only 1.67 minutes to administer, and has very good reliability (agreement 78%, κ=0.72, and weighted κ [κw]=0.82). Because the agreement was weakest on the questions related to mRS scores 3 to 5 (63% to 68%), we then tried to improve the smRSq by making the 2 questions regarding mRS scores 3 to 5 more specific. In this study, we tested a slightly revised smRSq for interrater agreement in person and by telephone and for correlation with a quality of life scale, the Short-Form-12 version 2 (SF-12v2).7
Patients and Methods
To improve the objectivity of the original smRSq,6 we made the questions related to scores 3 to 5 more specific (Figure; Table 1). We defined the ability to walk without assistance (mRS ≤3) as able to walk without assistance from one room to another in the patient's place of residence. We also defined not being bedridden (mRS ≤4) as able to go from lying to sitting under own power.
We screened patients for this study consecutively in 4 weekly clinics staffed by 4 stroke specialists and in 3 weekly resident continuity clinics at the Medical College of Georgia Hospital between July 2010 and October 2010. Included were patients with a diagnosis of ischemic or hemorrhagic stroke 1 to 12 months before enrolment who have been discharged from acute care. We confirmed the diagnosis of stroke if there was a sudden focal neurological deficit lasting >24 hours and cerebral imaging showed no other cause.
The 10 mRS Web-certified8 raters included 4 faculty stroke specialists, 3 neurology residents, 2 medical students, and a stroke research coordinator. For each subject, 3 raters were identified consecutively as first, second, and third from an alphabetic list of all 10 raters. The first 2 raters administered the smRSq in person within 20 minutes of each other, and the third rater administered the questionnaire by telephone 1 to 3 days later. Patients were instructed to respond to each interview session independently of the others. The third rater also administered the SF-12v2 by telephone,9 when possible. The SF-12v2 is a quality-of-life scale derived from the longer SF-36 version,7 consists of physical and mental components, and has been validated by telephone.9 When a designated rater was unavailable, the next rater on the list was contacted until an available rater was found. For each new patient, Rater 1 followed directly Rater 3 for the preceding patient on the alphabetic list.
The in-person ratings were done with patients and their caregivers when possible by asking the smRSq questions from top to bottom (Figure). In cases of disagreement between patients and their caregivers, the caregivers' answers were accepted as more accurate.10 The telephone smRSq was directed at the patients if their answers in person were not corrected by their caregivers. Otherwise, the telephone questionnaires were directed at the caregivers who assisted with the in-person interviews. Each rater's scores were concealed in envelopes from the other raters until all subjects were rated.
The SF-12v2 assessment was approved and added to this study after the first 11 patients were already scored and was administered only to the patients according to the scale instructions. Thus, some patients with cognitive deficits could not be scored with this instrument.
After rating all 50 patients in this study, the results were tabulated and analyzed using SAS statistical software (Cary, NC). We calculated the percentage agreement and determined κ and the κw for agreement between each pair of raters. Biserial correlation analysis compared the physical and the mental component SF-12v2 scores with the smRSq scores. Because each patient had 3 smRSq scores, the majority score was taken in cases of disagreement as the final score for this correlation. In cases of 3 different smRSq scores, the middle score was taken. Also, each rater estimated the average time they took to score the smRSq. This study was approved by the Medical College of Georgia Institutional Review Board.
The 50 patients in this study had a mean age of 61±14 (SD) years, 24 (48%) were men, 27 (54%) were black, 22 (44%) were white, and 1 (2%) was Asian. Eight patients (16%) had intracerebral hemorrhage and 42 (84%) had ischemic stroke. Average time from stroke to assessment was 4.83±3.00 (SD) months. The 4 faculty members performed a total of 62 ratings, the 3 residents 42, the 2 medical students 26, and the stroke research coordinator 20. The average time estimated by the 10 raters to administer the smRSq was 1.29 minutes (range, 0.5 to 2.25 minutes).
The in-person raters agreed 78% of the time with κ=0.71 (95% CI, 0.57 to 0.86) and κw taking into account the extent of disagreement=0.86 (0.79 to 0.94). The first in-person and the telephone raters agreed 82% of the time with κ=0.76 (0.63 to 0.90) and κw=0.87 (0.79 to 0.95). The second in-person and the telephone raters agreed 82% of the time with κ=0.77 (0.63 to 0.90) and κw=0.89 (0.82 to 0.96). There were insufficient rater pairs with similar levels of clinical experience for a meaningful reliability analysis based on experience. However, greater clinical experience did not seem to increase smRSq reproducibility because the agreement was 65% among the 17 ratings where each rater was faculty.
Table 1 shows the agreements for the 3 rater pairs on each of the smRSq questions. Table 2 shows the crosstabulation of the smRSq scores for the 3 rating pairs. The SF-12v2 scores were available for correlation analysis in 31 patients and in this subgroup, the smRSq agreements were similar to that of the entire study sample (κw=0.79 to 0.86). Patients with higher smRSq scores were more likely to have a lower quality of life (lower SF-12v2 scores). The correlation between smRSq and the SF-12v2 was better for the physical component (r=−0.50, P=0.005) than the mental component (r=−0.36, P=0.048).
We tested a novel slightly revised simplified mRS scoring method, the smRSq (Figure), for reproducibility after stroke, both in person and by telephone. In a subgroup of patients, we also compared the smRSq with a telephone validated quality-of-life scale, the SF-12v2.
The option to do a telephone assessment of stroke outcomes is important because some patients may not be able to return for a final assessment but may be available by telephone. A review of recent stroke trials showed that final outcome (usually the mRS) was assessed by telephone in 13% of 126 trials but possibly more because the method of outcome data collection was not described in 73% of the trials.2 Although reliability of the mRS by 2 telephone raters has not been reported, an in-person and telephone ratings have been compared.4,5 The agreement between telephone and in-person ratings using the unstructured mRS after ischemic stroke was excellent (κw=0.82) in 1 study,4 and using the structured mRS after subarachnoid hemorrhage was somewhat weaker (κw=0.71).5 In the present study, the smRSq between in person and by telephone compares somewhat better (κw=0.87 to 0.89) than the mRS in the previous studies.
Although only a subgroup of 31 patients had the SF-12v2 done, the smRSq agreements in this subgroup (κw 0.79 to 0.86) were similar to that of the entire study sample, suggesting similarity between these 2 populations. Other studies found significant correlations between the mRS and SF-36 (the parent instrument to the SF-12v2) after ischemic stroke,11–14 and 1 study found a significant correlation between the mRS and SF-12 after subarachnoid hemorrhage.15 In those studies, the mRS correlated better with the physical than the mental component of the quality-of-life scales, as might be expected from the physical construct of the mRS. In our study, the smRSq also correlated better with the physical than the mental component of the SF-12v2. The similar correlations with the quality-of-life scales in the previous studies and in our study suggest similarity between the smRSq and traditional mRS.
It is noteworthy that the agreements are ≥94% on the first smRSq question (Table 1), which is a common mRS primary outcome cut point (mRS ≤2) in clinical trials. Also, agreements are ≥86% on the question distinguishing between scores 2 and ≤1, which is another common primary outcome cut point. The agreements on the revised questions about walking (≥92%) and about being bedridden (≥75%) are better than in the original smRSq (68% and 63%, respectively).6
Repeat testing with the smRSq may have introduced recall bias. Such bias might be expected to be greater for the in-person ratings done within 20 minutes of each other than for the telephone ratings done 1 to 3 days later. We tried to limit such bias by instructing patients to treat each interview independently of the others. Also, some patients may have reconsidered and changed their answers between interviews, thus counteracting recall bias.
With the smRSq, we intended to create a simplified and more reliable method of scoring the mRS without altering its established validity. However, the smRSq method may yield somewhat different results than the traditional mRS,16 especially at the higher scores in which specific definitions about walking and being bedridden were introduced. However, because the ability to walk without assistance is a key distinction between mRS 3 and 4, the smRSq and the traditional mRS may yield similar results at this level. A direct comparison of the smRSq to the traditional or structured mRS, especially by other investigators, is needed and could identify differences between them, if present.
The slightly revised smRSq tested in this study shows excellent reliability that is similar to the original version6 and the structured interview mRS.3 In addition, this latest smRSq appears to have excellent reproducibility by telephone, correlates with a validated quality-of-life scale (SF-12v2),7 takes <1.5 minutes to administer, and can be used by a wide variety of raters. These features qualify it as a useful method of scoring the mRS.
- Received January 6, 2011.
- Revision received February 1, 2011.
- Accepted February 2, 2011.
- © 2011 American Heart Association, Inc.
- van Swieten JC,
- Koudstaal PJ,
- Visser MC,
- Schouten HJ,
- van Gijn J
- Quinn TJ,
- Dawson J,
- Walters MR,
- Lees KR
- Bruno A,
- Shah N,
- Lin C,
- Close B,
- Hess D,
- Davis K,
- et al
Modified Rankin Scale, English. Available at: http://trials-rankin.Trainingcampus.Net/uas/modules/trees/windex.Aspx. Accessed January 26, 2011.
- Rubenach S,
- Anderson CS,
- Laubscher S
- Knapp P,
- Hewison J
- Weimar C,
- Kurth T,
- Kraywinkel K,
- Wagner M,
- Busse O,
- Haberl RL,
- et al
- Quinn T,
- McArthur K,
- Dawson J,
- Walters M,
- Lees K