Validating the Questionnaire for Verifying Stroke-Free Status (QVSFS) by Neurological History and Examination
Background and Purpose— The Questionnaire for Verifying Stroke-Free Status (QVSFS) is an 8-item structured interview designed to identify stroke-free individuals. Previously, the QVSFS was validated with medical record review in a cohort with a low prevalence (7.1%) of stroke or transient ischemic attack (TIA). The objective of this study was to evaluate the validity of the QVSFS by comparing it with stroke status as determined by neurological history and examination in a population with a higher prevalence of stroke.
Methods— A research assistant administered the QVSFS to outpatients from Veterans Administration stroke and general medicine clinics. Subjects were defined as QVSFS negative if responses to all 8 questions were negative. Questions requiring rephrasing or clarification were noted. Neurologists, blinded to QVSFS scores, interviewed and examined all subjects to determine stroke-free status, defined as no history or examination findings of previous stroke and/or TIA.
Results— One hundred fifty-five subjects were examined; mean age was 70 years; 98.1% were male. Seventy-eight subjects were determined to be stroke free by the neurologist. The negative predictive value of the QVSFS was 0.96, with positive predictive value of 0.71. No question required rephrasing or clarification >5 times. Twenty-two subjects (14.2%) required rephrasing or clarification of at least 1 question.
Conclusions— The QVSFS can effectively identify stroke-free individuals with a high degree of accuracy, even in a population with a large proportion of patients with prior stroke or TIA. Accuracy for identifying subjects with stroke and/or TIA is lower, but the QVSFS may still be useful as a screening tool in that regard.
Accurate assessment of stroke status is fundamental to clinical stroke research. This is particularly true of epidemiological or genetic studies in which the association between a particular clinical or genetic variable and stroke is being examined. To date, genetic association studies in stroke typically have not specified how stroke-free status was verified or state simply that the control group consisted of “healthy blood donors.”1,2 History and physical examination and review of medical records are options for assessing stroke status; however, record reviews may be misleading since stroke may have occurred after the last record entry and because the absence of a particular disease may not be reliably recorded in the medical record. Furthermore, physical examination may be impractical. For example, in SWISS (Siblings With Ischemic Stroke Study), siblings of stroke patients will be geographically dispersed and may therefore be unavailable for examination by a study neurologist.3 For such subjects the most practical method of determining stroke status is to use an appropriately designed questionnaire. The Questionnaire for Verifying Stroke-Free Status (QVSFS), an 8-item structured questionnaire, was designed to quickly and accurately identify prospectively screened study subjects who are free of symptomatic cerebrovascular disease. The QVSFS was previously reported to have high sensitivity (1.0) and specificity (0.86) for stroke, transient ischemic attack (TIA), or both.4 Importantly, the negative predictive value (NPV) of the QVSFS was found to be exceptionally high (1.0). The positive predictive value (PPV), however, was much lower (0.36). The previous study was performed in a population with a low prevalence of stroke (7.1%), and stroke status was validated by medical record review rather than by history and examination. The low prevalence of stroke in the original study population may in part account for the high NPV seen previously. The primary objective of this study was to further examine QVSFS test characteristics by comparing it with stroke status as determined by neurological history and examination in a population with a higher prevalence of stroke. Secondary objectives were to evaluate the characteristics of the individual items in the questionnaire and compare them with the overall instrument and to examine patients’ understanding of questionnaire items.
Subjects and Methods
Subjects were outpatients seen in general internal medicine or stroke specialty clinics at a tertiary Veterans Administration hospital. Any patient who presented for routine care and who was able to hear and understand speech was eligible. In the general internal medicine clinic, subjects were limited to those aged 60 years and older to more closely replicate an at-risk stroke cohort.5 Sequentially eligible patients presenting to the clinics were approached until accrual was complete. We expected a higher proportion of patients in the stroke clinic than in the general medicine clinic to have a diagnosis of stroke. However, not all stroke clinic patients have a stroke diagnosis. Similarly, some patients followed in the general medicine clinics will have had a prior stroke. We intentionally sampled twice as many patients from the general medicine clinics so that the prevalence of stroke in our total sample would be more representative of the true prevalence in this age group.
The research assistant approached patients in the clinic, explained the study, and obtained informed consent. The research assistant administered the QVSFS (Figure) to those patients who consented and also recorded demographic information, including age, sex, and educational level. During the same clinic visit, a neurologist blinded to the QVSFS results interviewed each patient and performed a neurological examination. Both the research assistant and study neurologist were unaware of the stroke status of participants at the time patients were offered enrollment in the study. Study subjects were not told the results of either the QVSFS or the neurologists’ determination of their stroke status.
QVSFS scores range from 0 (no questions positive) to 8 (all 8 questions positive). Subjects were considered QVSFS negative (stroke/TIA free) if their sum score was 0. If any of the 8 items were positive (sum score 1 to 8), the QVSFS was considered positive (not stroke/TIA free.) Any question that was answered “unknown” was scored as a negative response. The alternative approach, scoring all “unknown” responses as positive, would have changed the QVSFS outcome in 6 subjects. The research assistant prospectively recorded every time a question had to be rephrased or clarified for the patient. If the neurologist determined that the examination was abnormal, the neurologist stated whether the abnormality was due to stroke or due to another cause. The examining neurologist determined each subject’s stroke status (stroke, TIA, or neither) on the basis of the face-to-face interview and examination results. This determination, based on history and examination, was considered the benchmark for stroke status in this study.
Test characteristics of the QVSFS were compared with benchmark determination of stroke status by Fisher’s exact test. Subject characteristics were compared with Fisher’s exact test or t tests between stroke-free subjects and subjects who were not stroke free. Comparisons were done separately for stroke status defined by the QVSFS and for stroke status determined by the benchmark history and examination. Age and educational level of subjects were also compared with t tests between those who did and those who did not require rephrasing or clarification of items. Positive and negative likelihood ratios were calculated to assess the relationship of each item and the QVSFS overall to the benchmark stroke status determination. The study was approved by the local human subjects review board.
One hundred fifty-five subjects were enrolled in this study: 98 from the general internal medicine clinic and 57 from the stroke specialty clinic. Subject characteristics are shown in Table 1. The mean age (±SD) of the predominantly (98%) male cohort was 70±8 years, and the educational level was 12±3 years. Subjects determined by the examining neurologist to have had a stroke/TIA were significantly younger (68 versus 71 years; P=0.02) than those without stroke/TIA; however, no significant difference in age was noted between subjects on the basis of a positive versus a negative QVSFS score.
Seventy-seven (49.7%) of the 155 subjects were found to have stroke/TIA as determined by the benchmark (Table 2). Of these 77, the QVSFS was also positive in 75 (97.4%; P<0.0001). In the 78 subjects who were stroke/TIA free by the benchmark, the QVSFS was negative in 47 (60.4%). The overall sensitivity and specificity for the QVSFS were 0.97 (95% CI, 0.94 to 1.0) and 0.60 (95% CI, 0.49 to 0.71), respectively. The NPV (ie, the probability of correctly identifying a stroke/TIA-free patient) was 0.96 (95% CI, 0.90 to 1.0), and the PPV (ie, the probability of correctly identifying a patient with stroke/TIA) was 0.71 (95% CI, 0.62 to 0.79). The positive and negative likelihood ratios were 2.45 and 0.04, respectively. The positive likelihood ratio is the odds of a positive QVSFS result in an individual with stroke/TIA (sensitivity/1−specificity); the negative likelihood ratio is the odds of a negative QVSFS result in an individual with stroke/TIA (1−sensitivity/specificity).
All of the 6 subjects whose QVSFS status would have changed if “unknown” responses had been scored positive were stroke/TIA free by the benchmark (ie, false-positive by QVSFS). This would result in no change in the sensitivity. The NPV would decrease only slightly from 0.96 to 0.95, and the negative likelihood ratio would slightly increase from 0.04 to 0.05. Specificity and PPV would decrease from 0.60 to 0.53 and from 0.71 to 0.67, respectively, and the positive likelihood ratio would decrease from 2.45 to 2.05.
The proportion of subjects answering “no” to each of the 8 questions ranged from 60.0% for question 1 to 85.2% for question 6 (Table 3). None of the individual items performed as well as the overall QVSFS. The sensitivity of the individual items ranged from 0.22 to 0.79, while the specificity ranged from 0.78 to 0.99. Question 1 was the most sensitive (0.79) and specific (0.99) individual item. The least sensitive were questions 6 and 7 at 0.22, with specificity of 0.92 and 0.87, respectively. The least specific individual item was question 4 at 0.78 (sensitivity, 0.56). The PPV for the individual questions ranged from 0.63 to 0.98. Only questions 5 and 7 had lower PPV (0.66 and 0.63, respectively) than the QVSFS as a whole. Accordingly, the positive likelihood ratios for questions 5 (1.94) and 7 (1.72) were also lower than the overall QVSFS, while for the other 6 questions the positive likelihood ratio was higher (2.56 to 61.79). The NPVs for the individual items ranged from 0.53 to 0.83, while the negative likelihood ratios ranged from 0.21 to 0.89.
Twenty-two study participants (14.2%) needed ≥1 of the questions to be rephrased or clarified (Table 3.) Only 3 of the 22 needed >1 question to be rephrased or clarified. All of the questions required rephrasing or clarification at least once. Question 1 required rephrasing or clarification the most at 5 times, while question 7 was least likely to require rephrasing or clarification (once). The average educational level of the subjects requiring rephrasing or clarification was 11±3 years (versus 12±3 years for those not needing rephrasing or clarification; P=0.09.) The average age of the subjects requiring rephrasing or clarification was 68±8 years (versus 70±8 years for those not needing rephrasing or clarification; P=0.21.) Half (11/22) of the questions requiring rephrasing were answered positive after rephrasing. Ten of the 20 subjects requiring ≥1 question to be rephrased or clarified were determined to have stroke on the basis of the benchmark (none were determined to have had a TIA). Of the 5 subjects requiring rephrasing or clarifying of question 1, 3 needed stroke to be distinguished from a heart attack, 1 reported a “sun stroke,” and 1 reported a seizure. All 4 of those requiring rephrasing or clarification of question 2 needed to have TIA defined for them. Six participants required emphasis of “sudden” onset of symptoms (questions 3, 4, 5, 7, and 8). Along with needing the term sudden emphasized twice for question 5, cataracts, eye injury, and nonspecific, transient descriptions of visual disturbances accounted for all of the confusion regarding questions 5 and 6.
Our results confirm a high sensitivity (0.97) and NPV (0.96) for the QVSFS, only slightly lower than the previous report.4 Importantly, this is true despite studying a more heterogeneous sample with much a higher prevalence of stroke (49.7% versus 7.1%) and more chronic disease than the original study. The low negative likelihood ratio (0.04), which is stable with changes in prevalence,6 indicates that fewer than 1 in 25 individuals with stroke/TIA is likely to be labeled stroke/TIA free by the QVSFS. Furthermore, in our study we validated the QVSFS results by comparing it with an interview and examination by an experienced neurologist rather than by medical record review. History and examination by an experienced physician is commonly accepted as the benchmark for determining disease status7 and enhances confidence in the accuracy of the questionnaire for truly identifying stroke/TIA-free individuals. Therefore, we conclude that the QVSFS is accurate for the intended purpose of identifying subjects who are free of stroke, TIA, or both.
Another important aspect of validity is questionnaire and item comprehension in the intended population. We found that the QVSFS was easily understood by most subjects. Fewer than 1 in 7 of the participants needed to have ≥1 question rephrased or clarified, and fewer than 1 in 56 questions needed rephrasing or clarification. The average educational level of subjects requiring rephrasing or clarification was approximately 1 year less than that in those not requiring rephrasing or clarification, a difference that approached statistical significance. However, there was no significant difference in the age of those needing rephrasing or clarification. Collectively, mistaken understanding of stroke or TIA was the most common reason for needing rephrasing or clarification and accounted for nearly half (9 of 22; all 5 of question 1 and all 4 of question 2) of the questions requiring rephrasing or clarification. Emphasis on sudden onset of symptoms accounted for 6 questions requiring clarification. Five questions required rephrasing or clarification because of other ocular diseases or symptoms not related to stroke/TIA. Importantly, this level of comprehension was observed in a population of clinic patients rather than in a group of hospital volunteers who are likely to be healthier and less likely to have multiple symptoms than subjects recruited from the clinic.
Previous studies have used either a single question about prior stroke8–12 or a multiple-item questionnaire about previous stroke and stroke symptoms to identify stroke cases.13–18 Most of these studies did not report test characteristics such as sensitivity or specificity.8,10,11,13,14,16–18 In the reports that did, sensitivity and specificity were 69% and 98%, respectively, for one instrument and 81.8% and 85.7%, respectively, for the other instrument.9,15 One study did not report sensitivity or specificity but did report the true-positive rate as 67%.12 Berger and colleagues19 compared a single question with a multiple-item questionnaire of stroke symptoms and found, as we did, increased accuracy using the multiple-item questionnaire versus a single question about previous physician-diagnosed stroke to assess stroke status in a general population. The sensitivity of their complete questionnaire was 89.5%. They also suggested different methodological strategies depending on the purpose for which an instrument is being used: prevalence or incidence estimation, risk factor analysis, or selection of proper controls.
The purpose of the QVSFS, to identify individuals who are free of stroke/TIA, is most consistent with the strategy of selecting proper controls suggested by Berger et al.19 For this strategy they used 2 questions about impairment of vision and a question about previously diagnosed stroke, which resulted in a sensitivity of 84.2%, moderately lower than the overall sensitivity of 97% for the QVSFS. Another potential use of the QVSFS, given the low negative likelihood ratio of 0.04, is to assist in case ascertainment of subjects with stroke by efficiently reducing the number of subjects requiring a comprehensive medical record review and/or examination. Since fewer than 1 in 25 subjects with stroke/TIA would be labeled stroke/TIA free by the QVSFS, this may reduce the time and cost of accurately identifying stroke cases.
Although the QVSFS was accurate for identifying stroke/TIA-free patients in this study, some limitations remain. Recall bias and comorbidity may both reduce the sensitivity and specificity of a questionnaire if symptoms are incorrectly ascribed to stroke. To limit this potential bias in our cohort of chronically ill patients, we validated our results by physician history and examination, which most accurately classifies stroke/TIA status, and found that the sensitivity of the QVSFS remained very high. In addition, the performance characteristics of the QVSFS are not yet established for self-administration (for example, by postal survey). Additionally, the subjects in this study were predominantly men, although the original QVSFS validation sample included a more representative proportion of women and found very similar results. In addition, while valid for the purpose of identifying stroke/TIA-free individuals, the QVSFS may misclassify some individuals as having stroke/TIA when they are actually stroke/TIA free. The resulting implication for investigators is that, if they use the QVSFS to identify individuals with stroke, the presence of stroke must be confirmed by another means.
Accurate identification of stroke status is fundamental to many types of stroke research. The QVSFS provides a rapid, valid means of identifying stroke/TIA-free subjects. It may also be used to reduce the number of potential subjects screened with more labor-intensive methods for the purpose of case ascertainment. Work in SWISS will help to further confirm the generalizability across a heterogeneous sample and to establish the reliability of the QVSFS.
This study was supported by a Veterans Administration Ambulatory Care fellowship grant (Dr Jones); by a Veterans Administration Career Development award, Health Services Research Division (Dr Williams); and by National Institutes of Health/National Institute of Neurological Disorders and Stroke grant NS39987 (Dr Meschia). This work was done in part in the Regenstrief Institute for Health Care.
- Received March 7, 2001.
- Revision received June 1, 2001.
- Accepted June 28, 2001.
Carlsson LE, Santoso S, Spitzer C, Kessler C, Greinacher A. The alpha2 gene coding sequence T807/A873 of the platelet collagen receptor integrin alpha2beta1 might be a genetic risk factor for the development of stroke in younger patients. Blood. 1999; 93: 3583–3586.
Meschia JF, Brott TG, Chukwudelunzu FE, Hardy J, Brown RD, Meissner I, Hall LJ, Atkinson EJ, O’Brien PC. Verifying the stroke-free phenotype by structured telephone interview. Stroke. 2000; 31: 1076–1080.
Wolf PA, Cobb JL, D’Agostino RB. Epidemiology of stroke.In: Barnett HJM, Mohr JP, Stein BM, Yatsu FM, eds. Stroke: Pathophysiology, Diagnosis, and Management. New York, NY: Churchill Livingstone; 1992: 3–27.
Sackett DL, Haynes RB, Guyatt GH, Tugwell P. The interpretation of diagnostic data.In: Clinical Epidemiology: A Basic Science for Clinical Medicine. Boston, Mass: Little, Brown & Co; 1991; 69–152.
Kannel WB, Dawber TR, Cohen ME, McNamara PM. Vascular disease of the brain-epidemiologic aspects: the Framingham Study. Am J Public Health. 1965; 55: 1355–1366.
Aho D, Reunanen A, Aromaa A, Knekt P, Maatela J. Prevalence of stroke in Finland. Stroke. 1986; 17: 681–686.
Haapanen N, Miilunpalo S, Pasanen M, Oja P, Vuori I. Agreement between questionnaire data and medical records of chronic diseases in middle-aged and elderly Finnish men and women. Am J Epidemiol. 1997; 145: 762–769.
Colditz GA, Martin P, Stampfer MJ, Willett WC, Sampson L, Rosner B, Hennekens CH, Speizer FE. Validation of questionnaire information on risk factors and disease outcomes in a prospective cohort study of women. Am J Epidemiol. 1986; 123: 894–900.
Bergmann MM, Byers T, Freedman DS, Mokdad A. Validity of self-reported diagnoses leading to hospitalization: a comparison of self-reports with hospital records in a prospective study of American adults. Am J Epidemiol. 1998; 147: 969–977.
O’Mahony PG, Dobson R, Rodgers H, James OFW, Thomson RG. Validation of a population screening questionnaire to assess prevalence of stroke. Stroke. 1995; 26: 1334–1337.
Urakami K, Igo M, Takahashi K. An epidemiologic study of cerebrovascular disease in Western Japan: with special reference to transient ischemic attacks. Stroke. 1987; 18: 396–401.
Toole JF, Lefkowitz DS, Chambless LE, Wijnberg L, Paton CC, Heiss G. Self-reported transient ischemic attack and stroke symptoms: methods and baseline prevalence: the ARIC Study, 1987–1989. Am J Epidemiol. 1996: 144: 849–856.
Walker MK, Whincup PH, Shaper AG, Lennon LT, Thomson AG. Validation of patient recall of doctor-diagnosed heart attack and stroke: a postal questionnaire and record review comparison. Am J Epidemiol. 1998; 148: 355–361.