Postal and Face-to-Face Administration of Stroke Outcome Measures
Can Mixed Modes Be Used?
Background and Purpose—Different modes of administration are used to collect stroke outcomes, even within the same study, potentially leading to different results. We investigated the effect of administration mode (postal questionnaire; face-to-face interview) on self-reports of activities of daily living and mood.
Methods—The study was nested within a poststroke motivational interviewing trial. Activities of daily living (Barthel; Nottingham Extended) and mood (General Health Questionnaire; Yale) were collected at 3 and 12 months via postal questionnaire. Participants were approached to respond again via face-to-face interview. Paired t tests (McNemar test) and intraclass correlation coefficients (Cohen κ) were used, with 95% CI, to compare scores (items).
Results—Forty-four participants consented. Only Barthel scores were significantly different; they were 1.0 (95% CI, 0.5–1.6) higher face-to-face. The intraclass correlation coefficient for the Barthel was 0.90; for the other scales it was between 0.83 and 0.87. The Yale κ was 0.72.
Conclusions—Modes of administration might be used interchangeably, albeit in conjunction with corrections for the Barthel.
Stroke recovery is affected by limitations in activities of daily living (ADL) and mood,1 and stroke treatment and care evaluations include measures of function and emotion. Validated questionnaires are used because collection of directly observed performance measures is often impractical and direct observation of mood has limited success.2 However, delivery mode can affect their measurement properties. Question meanings can be clarified in face-to-face interviews, but these are expensive and interviewers may misinterpret responses.3 Postal administration is cheaper and perceived anonymity may enhance sensitive information disclosure, but questionnaire completion may be compromised.3 Stroke adds further complexity: patient interpretation or communication is often compromised and upper limb hemiplegia makes self-completion difficult. We found no studies comparing postal and interviewer administration for poststroke mood outcomes and few for other constructs.4,5 This study compares ADL and mood responses from postal questionnaire and face-to-face interview, and explores factors which may explain response differences.
The study was conducted within a poststroke motivational interviewing randomized controlled trial.1 Ethical approval was obtained. Baseline measures, collected face-to-face by a research nurse, included cognition (Rivermead Behavioural Memory Test),6 communication (Frenchay Aphasia Screening Test),7 mood (12- and 28-item General Health Questionnaire8; Yale),9 and function (Barthel Index).10 Outcomes collected 3- and 12-months poststroke by postal questionnaire included General Health Questionnaire-12, General Health Questionnaire-28, Barthel Index,5 Nottingham Extended ADL,11 and Yale.
Trial participants returning 3- or 12-month postal questionnaires between January 2005 and February 2006 were flagged, without disclosing their data, by a research assistant to an experienced research nurse. Those recruited at 3 months were not approached again at 12 months. The nurse arranged a timely face-to-face interview (within 14  days of questionnaire receipt for 3--month follow-up) and read out questions identical to the postal questionnaire, recording participants’ responses.
Sample size target was 50 so that 95% CIs for intraclass correlation coefficients of scale scores would have width of <0.2, assuming intraclass correlation coefficient is ≥0.8. Differential bias was assessed using 95% CI based on paired t tests (McNemar test for Yale). Repeatability was measured using intraclass correlation coefficient for scales and Cohen κ for items (with quadratic weighting when ordinal), with 95% CI, using bias-corrected bootstrapping for κ. Scale scores were compared using Bland-Altman and scatter plots. For scales, if repeatability was inadequate we performed item analysis, and if differential bias was detected we used exploratory linear modeling to investigate whether participant characteristics could explain between-mode differences.
Postal questionnaires were returned by 109 trial participants during the recruitment period, of whom 44 (40.4%) were interviewed (Table 1).
Only the Barthel showed a significant between-mode difference, with mean scores 1.0 (95% CI, 0.5–1.6) higher face-to-face (Table 2); differences appeared larger among participants with lower scores (Figure). Repeatability was generally high (Table 2).
Differential bias in Barthel scores was 2.0 (95% CI, 0.8–3.2) higher in those with baseline communication problems and increased by 0.5 (0.0–0.9) for each 5 points lower in baseline Barthel score; effects of stroke type (P=0.44), side (P=0.84), cognition (P=0.20), outcome time point (P=0.83), and intervention group (P=0.28) were nonsignificant.
This study suggests consistency between stroke patient mood and ADL questionnaire responses postally and face-to-face. However, Barthel scores were often higher face-to-face, resulting in a 1-point significant difference in means. People may be more likely to self-report disability postally, suspecting it might affect benefit entitlement; social desirability bias might affect face-to-face interviews.3 Potentially embarrassing questions around incontinence showed relatively large biasing effects, albeit of borderline significance. Some differences between delivery modes have previously been found in item responses on other ADL questionnaires.4 Seven postal questionnaires were completed with support from carers; proxies have been shown to rate outcome worse than stroke patients.12
Barthel delivery mode could affect conclusions of disability surveys, but in randomized controlled trials its influence would not differ between arms, unless assessment bias was different; there was no intervention group effect on between-mode differences in our study. Alternatively, assessment bias may result in ceiling effects, a recognized problem with the Barthel,13 potentially reducing treatment effects; we found smaller between-mode differences in participants near its ceiling. If face-to-face interviewing is used to supplement postal collection, it would be prudent to consider potential biases.
Our study is not without limitations. Postal questionnaires were always completed first, but most participants were 12-months poststroke and follow-up time was not related to differential bias, so functional improvement is unlikely to explain our findings. Most participants had high Barthel scores and, because of the trial intervention, no patient had severe cognitive or communication problems. Although our sample was small and homogeneous, we detected factors that might help explain Barthel differences. We were unable to conclude which mode is more valid as there was no objective ADL assessment.
We therefore recommend further research, using larger, more comprehensive samples, and randomized design, into using different delivery modes for the Barthel, including telephone interviewing not investigated here. Research should include investigating corrections for potential differential bias introduced via mixed assessment methods. In the interim, care should be taken in mixing modes of Barthel administration, particularly when differential bias cannot be reasonably assumed to cancel out.
The authors acknowledge Helen Gardner and Irene McClelland (data collection), Denise Forshaw (data coordination/input), and Jenny Marsden (data management).
Sources of Funding
This study was funded by National Health Service (NHS) Executive, North West; Royal Liverpool and Broadgreen University Hospitals NHS Trust; and University of Central Lancashire.
- Received July 20, 2012.
- Revision received September 29, 2012.
- Accepted October 4, 2012.
- © 2012 American Heart Association, Inc.
- Watkins CL,
- Auton MF,
- Deans CF,
- Dickinson HA,
- Jack CI,
- Lightbody CE,
- et al
- Bowling A
- Gompertz P,
- Pound P,
- Ebrahim S
- Wilson B,
- Baddeley AD,
- Cockburn J
- Nouri FM,
- Lincoln NB
- Carod-Artal FJ,
- Ferreira Coral L,
- Stieven Trizotto D,
- Menezes Moreira C
- Quinn TJ,
- Langhorne P,
- Stott DJ