Improving the Assessment of Outcomes in Stroke: Use of a Structured Interview to Assign Grades on the Modified Rankin Scale
To the Editor:
We read with interest the article by Wilson et al,1 describing the use of a structured interview for the modified Rankin Scale (mRS). We attempted a similar exercise to validate the assessment of the mRS over the telephone by nonphysician interviewers using an analogously structured interview.
Thirty-four patients with a discharge diagnosis of stroke participated in the study 3 to 6 months from the day of admission. All patients were seen in person first, where they provided informed consent, by two trained personnel (one stroke neurologist or one of three stroke nurses). Each patient was interviewed consecutively by two interviewers in the Stroke Prevention Clinic or, for badly disabled patients, at a current place of residence (long-term care facility, nursing home, or rehabilitation facility) and the mRS assigned. Each face-to-face interview was followed by a telephone interview conducted by trained interviewers following a structured questionnaire within 5 days of the original interview. No proxy interviews were used. Each person rated the patient on the mRS, blind to the other raters’ score. Overall, three observers scored each patient’s outcome.
Two telephone interviewers were not medical professionals (RN or MD) but had 5 or more years of experience with telephone interviewing, had completed a course in telephone interviewing, and were considered experienced. Questions were scripted and deviations from the script were not permitted. The study was approved by the local Ethics Review Board.
Agreement, assessed using unweighted κ statistics with 95% confidence limits, was excellent between the two observers in the stroke clinic (κ=0.72 [0.55 to 0.89]), but less impressive between either clinical observer and the telephone interviewer (κ=0.38 [0.21 to 0.55] and κ=0.30 [0.13 to 0.47]).2 The telephone interviewer was most likely to agree with the in-clinic observer at the extremes of the scale (mRS=0 or mRS≥4). In the mid-range where the scale is commonly dichotomized (0 to 2 versus 3 to 5 and 0 to 1 versus 2 to 5), the telephone interviewer correctly rated the patient only 50% of the time.
Predictably, a structured interview for the mRS improves reliability between observers. However, our experience suggests that telephone assessment of the mRS may result in substantial variability in scoring. It remains important for physicians and/or trained nurses to score patients in person rather than relying on telephone interviewers.
Telephone interviews are a popular method of assessing outcome after stroke because of their low cost and simplicity. Studies have indicated that information concerning activities of daily living after stroke can be reliably obtained by telephone.1,2 In contrast, Hill and colleagues found low interobserver agreement for the modified Rankin Scale (mRS) when using a structured questionnaire.
The report sounds an appropriate cautionary note and serves as a reminder that it has not been fully established what contributes to the success or failure of telephone interviews. For example, it is not clear whether it is better to adhere to a script or allow rephrasing of questions when there appear to be communication problems. Telephone interviews tend to elicit shorter, less informative responses than face-to-face contact, and this may contribute to disagreement between interviewers. Use of skilled interviewers, as here, should help to minimize these problems. However, in addition to standardizing the questions, we specified the criteria that are used to interpret the responses.3 The midrange of the mRS is particularly problematic in this respect, because the categories are not well defined and therefore open to rater interpretation. Differences in interpretation are more likely when interviewers come from different professional backgrounds, and this may also contribute to the low agreement found by Hill et al. Previous work shows that differences in observers may be more important than differences in modality.1
We have not yet specifically tested the modified Rankin Scale structured interview (mRS-SI) in a telephone interview, but we hope that the significantly improved interrater reliability of the mRS-SI would be reproduced in this setting. By specifying the definition of each level of the mRS, the mRS-SI removes a large part of the subjectivity that blights distinction between shades of disability. Further work has confirmed that the benefits of the mRS-SI are more striking when raters from varied backgrounds are involved.
Candelise L, Pinardi G, Aritzu E, Musicco M. Telephone interview for stroke outcome assessment. Cerebrovasc Dis. 1994; 4: 341–343.
Wilson JTL, Hareendran A, Grant M, Baird T, Schulz UGR, Muir KW, Bone I. Improving the assessment of outcomes in stroke: use of a structured interview to assign grades on the modified Rankin Scale. Stroke. 2002; 33: 2243–2246.