Development and Validation of a Bilingual Stroke Preparedness Assessment Instrument
Background and Purpose—Stroke preparedness interventions are limited by the lack of psychometrically sound intermediate end points. We sought to develop and assess the reliability and validity of the video-Stroke Action Test (video-STAT) an English and a Spanish video-based test to assess people’s ability to recognize and react to stroke signs.
Methods—Video-STAT development and testing was divided into 4 phases: (1) video development and community-generated response options, (2) pilot testing in community health centers, (3) administration in a national sample, bilingual sample, and neurologist sample, and (4) administration before and after a stroke preparedness intervention.
Results—The final version of the video-STAT included 8 videos: 4 acute stroke/emergency, 2 prior stroke/nonemergency, 1 nonstroke/emergency, and 1 nonstroke/nonemergency. Acute stroke recognition and action response were queried after each vignette. Video-STAT scoring was based on the acute stroke vignettes only (score range 0–12 best). The national sample consisted of 598 participants, 438 who took the video-STAT in English and 160 who took the video-STAT in Spanish. There was adequate internal consistency (Cronbach α=0.72). The average video-STAT score was 5.6 (SD=3.6), whereas the average neurologist score was 11.4 (SD=1.3). There was no difference in video-STAT scores between the 116 bilingual video-STAT participants who took the video-STAT in English or Spanish. Compared with baseline scores, the video-STAT scores increased after a stroke preparedness intervention (6.2 versus 8.9, P<0.01) among a sample of 101 black adults and youth.
Conclusions—The video-STAT yields reliable scores that seem to be valid measures of stroke preparedness.
The overwhelming reason that stroke patients do not receive acute stroke treatment is their late arrival to the hospital, outside the acute stroke treatment window.1 Calling 911 is associated with a decrease in time from stroke symptom onset to arrival to the emergency department.2 Therefore, stroke preparedness interventions, defined as interventions that aim to increase stroke symptom recognition and 911 calls, are critically important to increasing intravenous tissue-type plasminogen activator administration and mechanical thrombectomy and reducing disability after stroke.
Despite the high incidence of strokes in the United States, the probability that any 1 person will witness a stroke is low. Thus, trials that have assessed clinical end points such as tissue-type plasminogen activator treatment rates or decreased prehospital delay have primarily intervened on communities.3,4 These trials require substantial resources and cost. In addition, entire communities are exposed to a potentially inefficacious intervention, diverting resources from other more promising interventions or initiatives. Preliminary studies with intermediate end points can help improve the efficiency of community-level trials by helping to identify promising interventions. Intermediate end points also facilitate efficient intervention testing for selected populations, including high stroke risk groups such as blacks and Hispanics.5,6
However, there is a lack of a psychometrically sound stroke preparedness intermediate end point. Currently, stroke preparedness instruments, such as the stroke action test (STAT), rely on written descriptions of stroke, which ignore the auditory and visual experience of witnessing and reacting to stroke.7–10 Video-based tests could provide a more realistic stimulus and would therefore be expected to provide a more accurate assessment of whether someone would in fact recognize and act in real life.11 In other disease areas, behavioral intent elicited from video vignette has been found to closely mirror actual behavior in live situations, a necessary characteristic of intermediate outcomes.12–14 In partnership with community health centers, we developed a video-based test, video-STAT, to assess people’s ability to recognize and react to stroke signs. The video-STAT comprised simulated patient video vignettes in English and Spanish. This article describes test development and assessment of reliability and validity of the video-STAT.
Video-STAT development was divided into 4 phases: (1) video development and community-generated response options, (2) pilot testing in community health centers, (3) administration in a national sample, bilingual sample, and neurologist sample, and (4) administration before and after a pilot stroke preparedness intervention (Table 1).
Phase 1: Video Vignette and Post-Vignette Question Development
To minimize respondent burden and stay within the constraints of the available resources, 10 video vignettes scripts were drafted by vascular neurologists. The vignettes included 5 acute stroke/emergency, 2 prior stroke/nonemergency, 1 nonstroke/emergency (acute myocardial infarction), and 2 nonstroke/nonemergency scenarios (Table I in the online-only Data Supplement). All vignette scripts were drafted in English, underwent Spanish translation, and were then translated back into English to check translation accuracy. Prior stroke/nonemergency and nonstroke vignettes were included to avoid creating a measure where the correct response was always to call 911. The scripts underwent content review for clinical accuracy and community appropriateness by a team of stroke experts, including vascular neurologists, general neurologists, emergency department physicians, stroke nurses, stroke study coordinators of the University of Michigan, and community health center providers and staff. The scripts were edited based on this expert feedback. The vignettes were then filmed by a professional educational video production company. A vascular neurologist and a health education expert assisted in the selection of the actors and were present for all filming. In addition, postvignette questions, with various phrasing of the questions, were drafted. These questions queried (1) stroke recognition and discrimination and (2) action response (eg, call 911, drive to emergency department).
After the vignette filming, cognitive interviews were conducted at 3 community health centers which serve large Hispanic and black patient populations. Partnership with community health centers helped to ensure adequate representation of racial/ethnic minorities and those with low socioeconomic status, populations with disproportionate stroke risks. Two sets of cognitive interviews were performed. The first set included 11 participants and was performed after filming the first 3 videos to ensure community appropriateness and realism of the videos before filming all the videos. The second set included 24 participants who viewed and responded to all 10 video vignettes. The goals of the cognitive interviews were to (1) assess cultural appropriateness, (2) determine the optimal phrasing of the postvignette stroke recognition and action response questions, and (3) generate response options to the postvignette questions that incorporated wording and content provided by community members. The interviews lasted about 30 minutes. Twenty dollars was given to each participant who completed the interview.
Phase 2: Video-STAT Community Pilot Testing
The video-STAT was pilot tested with 82 participants recruited from community health centers, including 9 Spanish-speaking participants. It was administered by a member of the research team during a face-to-face interview, and participants’ responses to the vignettes were recorded. Twenty dollars was given to each participant after interview completion.
Phase 3: Testing in National, Bilingual, and Neurologist Samples: Video-STAT Establishment of Scoring, Assessment of Reliability, and Validity
Testing in a Nationally Representative Sample
The video-STAT was administered to a nationally representative sample randomly drawn from GfK Knowledge Panel, an online research participant panel.15 The Knowledge Panel is recruited using address-based sampling, which covers 97% of the US population and currently contains 55 000 adult members.15 Computer and internet access are provided to those participants without access. The video-STAT was administered to 598 participants divided among non-Hispanic whites, non-Hispanic blacks, Hispanics who are English-language survey-takers, and Hispanics who are Spanish-language survey-takers. Five-dollar cash-equivalent incentives were given to participants.
Language equivalency was tested by randomizing 116 bilingual Knowledge Panel participants to complete the video-STAT in English or Spanish.
The video-STAT was administered to a bilingual vascular neurologist to ensure equivalent representation of the English and Spanish vignettes. In addition, an e-mail link was sent to about 55 attending academic neurologists of which 14 (25%) completed the video-STAT.
Phase 4: Testing in a Stroke Preparedness Pilot Intervention
The research team conducted a peer-led, workshop-based, health behavior intervention to increase stroke preparedness among 101 black youth and adults in Flint, MI.16 Participants were recruited from 3 predominately black churches. The intervention consisted of 2 peer-led workshops separated by 1 week. The video-STAT was used to assess changes in stroke preparedness before workshop 1 (baseline) and after workshop 2 (post-test). Although the same videos were used in the baseline and post-test, the community members were not informed of the correct answers before the post-test.
Video-STAT Vignette Selection and Scoring
On the basis of the results of the community pilot testing, national testing, bilingual testing, and neurologist testing, 8 out of the 10 video vignettes were retained for scoring. One nonstroke, nonemergency video that portrayed a mechanical fall was eliminated after pilot testing because of poor discrimination (all participants correctly responded to the vignette). Second, there was a lack of consensus among the neurologists as to the correct response to the final video, showing a black woman with acute gait ataxia. Therefore, this vignette was dropped (Table I in the online-only Data Supplement).
To emphasize the importance of behavioral intent, the action response was weighted more heavily than stroke recognition (Table I in the online-only Data Supplement). In the acute stroke vignettes, there was a maximum score of 1 point for stroke recognition and 2 points for the correct action of calling 911. For nonacute stroke vignettes, 1 point was given for any recognition answer other than acute stroke, and 2 points were given for the action response of watchful waiting with the exception of the prior stroke vignette that had not undergone evaluation (Table I in the online-only Data Supplement; vignette number 7). The myocardial infarction vignette action was scored similar to the acute stroke vignettes with 2 points for the correct action of calling 911. The recognition and action items of each vignette were summed to obtain a 0 to 3 point scale for each vignette.
Total video-STAT scores were calculated as the sum of the scores for each of the retained acute stroke vignettes, each of which ranges from 0 to 3 points per vignette (described above), resulting in a total score ranging from 0 to 12. Various scoring rules were considered, including the possibility of adding points across all videos retained. The final scoring rule was informed by clinical expertise, results of factor analysis, and reliability and item statistics (described below).
Descriptive statistics were used to assess the sociodemographics of the national sample. Exploratory factor analysis was performed in the overall sample and separately according to the language of administration to assess if the factor pattern was similar across languages. The number of factors was chosen based on eigenvalue criteria and examining the scree plot. Items were determined to load on a factor if their factor loadings were at least 0.40. Reliability was assessed using Cronbach α. A reliability of ≥0.70 is considered sufficient for instruments where the primary goal is to assess differences in group means, as is likely to be the case in future intervention studies.17 The mean score was calculated for each vignette and for the combined total score among the national sample, as well as separately by the language of administration among the bilingual sample. Among the bilingual sample, differences in the average score between Spanish- and English-language test-takers were conducted with a t test. Item means and total scores were similarly calculated among the neurologist sample and among the participants in the pilot intervention. Among the pilot intervention participants, differences in baseline and postintervention video-STAT scores were assessed using a paired t test. Statistical analyses were conducted in SAS (version 9.3) and STATA (version 11.0). This study was approved by the University of Michigan Internal Review Board.
The video-STAT was administered to 714 Knowledge Panel participants of which 598 were from the national sample, and 116 were from the bilingual sample. The video-STAT is available in English at https://youtu.be/89-biRONpCk and in Spanish at https://youtu.be/9X6pe7oP8cI. The video-STAT was completed in a median of 16.2 minutes (interquartile range, 14.2–20.5 minutes).
National Sample Testing
Of the 598 participants in the national sample, 438 took the video-STAT in English and 160 in Spanish (Table 2). The mean age was 48 years; 50% of the English test-takers were women, whereas 56% of the Spanish test-takers were women. The English test-takers were evenly divided among whites, blacks, and Hispanics. Participants lived throughout the United States. Spanish test-takers reported lower income than English test-takers. Home internet access was provided to 13% of English test-takers and 16% of Spanish test-takers.
The video-STAT was administered to 116 bilingual participants, of whom 61 were randomized to the English version and 55 to the Spanish version. There was no difference in age, sex, or education between participants randomized to the English vignettes compared with those who completed the Spanish vignettes (Table 2). There was no difference in the total score of the video-STAT (5.8 English vignettes versus 5.3 Spanish vignettes; P=0.56). Reliability was adequate for both the English (α=0.75) and Spanish (α=0.72) versions of the video-STAT.
Factor Analysis, Reliability, and Item Correlations
Given that there were no significant differences among the bilingual sample, results for the overall sample were combined (ie, all 714 participants) for psychometric evaluation. Exploratory factor analysis showed that the video vignettes load on to 2 factors: (1) acute events (stroke and myocardial infarction) and (2) nonacute stroke, nonemergency (Table 3). All acute stroke vignettes loaded onto the acute factor, with loadings ranging from 0.57 to 0.76. The acute myocardial infarction vignette also loaded on this factor with a factor loading of 0.59. The prior stroke/nonemergency and nonstroke/emergency vignette had low or negative loadings on the acute factor (−0.05 to 0.11) but loaded more highly on a second factor of nonacute events with factor loadings 0.58 to 0.73. There was a small difference in reliability with or without the acute myocardial infarction vignette (Cronbach α=0.72 versus 0.71). Thus, based on clinical judgment, ease of use, and item reliability, the acute myocardial infarction vignette was dropped from the video-STAT score. Thus, the video-STAT score is based solely on acute stroke vignettes. Item-total correlations ranged from 0.39 to 0.55.
Table 4 shows the means and SDs for the acute stroke vignettes that comprise the scored portion of the video-STAT. The average total video-STAT score was 5.6 (SD=3.6), suggesting the absence of floor and ceiling effect. The differences between the English and Spanish test-takers are likely because of demographic differences between the groups.
The average score among neurologists was 11.4 (SD=1.3). A perfect score was obtained by 74% of the neurologists.
Sensitivity to Stroke Preparedness Intervention
A total of 101 black participants (73 adults and 28 youth) completed the baseline assessment, and 64 completed the post-test (47 adults and 17 youth). The median age of the adult participants was 56 (interquartile range, 35–65), 65% were female, and 38% had a high school education or less. The median age of the youth participants was 14 (interquartile range, 11–16), and 50% were female. Compared with baseline scores, video-STAT scores increased in the post-test (6.2 versus 8.9, P for paired t test <0.01).
The video-STAT is a new video-based measure of stroke recognition and action response available in both English and Spanish for use in assessing stroke preparedness of community members. The average time to complete the video-STAT is 16 minutes. Stroke experts created the vignettes, ensuring clinical accuracy and realism. Response options were community generated, ensuring that wording made sense to lay people, and that options were ones which community members might reasonably consider. The video-STAT scores were reliable and seem valid.
Establishing the score validity is a process. The careful instrument development process described here is the first step in this process. Administering the video-STAT to a nationally representative sample allowed us to gather additional validity evidence by examining the dimensionality responses and item-total correlations. It also allowed us to examine score reliability in a large, diverse sample of community adults. Our finding of no detectable difference in scores between randomly assigned bilingual Knowledge Panel participants who took the video-STAT in English compared with those who took it in Spanish is evidence of the equivalence of the 2 versions. Our finding that stroke experts’ scores were much higher than scores in the general population is evidence that the video-STAT is sensitive to the differences in stroke recognition and action response and is evidence of validity. Our finding that community members’ video-STAT scores increased after a pilot intervention intended to increase stroke preparedness is further validity evidence. Taken together, these results support the use of the video-STAT in future interventions to assess stroke preparedness.
The video-STAT was developed, pilot tested, and validated with a large ethnically and racially diverse sample from across the United States, including English- and Spanish-speaking participants. The ability to use the video-STAT to assess acute stroke interventions in diverse populations is a strength given the disproportionate burden of stroke among blacks and Hispanics.5,6 On completion of the development and validation process, we uploaded the video-STAT to YouTube for use by other groups assessing stroke preparedness. Future work could consider adapting the video-STAT to other languages, such as Mandarin, Hindi, or Arabic, which may require some modification to the sociocultural references.
This study has limitations. Although the video vignettes are more realistic than written vignettes or descriptions of stroke, viewing a video cannot simulate the emotions of an actual stroke scenario where the stroke patient may be a friend or relative. Although the video-STAT is available in both English and Spanish, it is not available in other languages and thus cannot be used in other populations.
In conclusion, our findings provide evidence that video-STAT scores are reliable and valid measures of stroke preparedness. Our results support the use of the video-STAT in conjunction with interventions to decrease prehospital delay or increase acute stroke treatment.
Sources of Funding
This work was supported by National Institutes of Health (NIH)/National Institute of Neurological Disorders and Stroke (NINDS) R21NS084081 and NIH/NINDS K23 NS073685 (Dr Skolarus)
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.116.015107/-/DC1.
- Received August 15, 2016.
- Revision received December 19, 2016.
- Accepted January 9, 2017.
- © 2017 American Heart Association, Inc.
- Kleindorfer D,
- Kissela B,
- Schneider A,
- Woo D,
- Khoury J,
- Miller R,
- et al
- Boden-Albala B,
- Edwards DF,
- St Clair S,
- Wing JJ,
- Fernandez S,
- Gibbons MC,
- et al
- Morgenstern LB,
- Staub L,
- Chan W,
- Wein TH,
- Bartholomew LK,
- King M,
- et al
- Kleindorfer D,
- Miller R,
- Sailor-Smith S,
- Moomaw CJ,
- Khoury J,
- Frankel M
- Morgenstern LB,
- Gonzales NR,
- Maddox KE,
- Brown DL,
- Karim AP,
- Espinosa N,
- et al
- Williams O,
- Noble JM
- Billings-Gagliardi S,
- Mazor KM
- McKinstry B
- 15.↵GFK. Knowledge Panel Design Summary. Available at: http://www.knowledgenetworks.com/ganp/docs/KnowledgePanel(R)-Design-Summary.pdf. Accessed May 13, 2016.
- Skolarus LE,
- Zimmerman MA,
- Bailey S,
- Dome M,
- Murphy JB,
- Kobrossi C,
- et al
- DeVellis RF