| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Stroke. 2000;31:858.)
© 2000 American Heart Association, Inc.
Original Contributions |
From Roudebush Veterans Affairs Medical Center (L.S.W.) and the Department of Neurology, Indiana University School of Medicine (L.S.W., E.Y., A.M.L.-Y.), Indianapolis, Indiana.
Correspondence to Linda S. Williams, M, Roudebush VAMC, HSR&D 11-H, 1481 West 10th Street, Indianapolis, IN 46202. lwilliams@hsrd.va.iupui.edu
| Abstract |
|---|
|
|
|---|
MethodsAn algorithm for retrospective NIHSS scoring was developed with written history and physical admission notes. Missing physical examination data were scored as normal. One investigator prospectively scored the admission NIHSS in 32 consecutive stroke patients. Two raters retrospectively scored the NIHSS by applying the algorithm to photocopied admission notes. Linear regression was used to assess interrater reliability and agreement between prospective and retrospective NIHSS scores. The Wilcoxon signed rank test was used to assess systematic scoring bias. Weighted kappa statistics were calculated to assess the level of agreement of individual NIHSS items.
ResultsOnly 1 admission note was complete for all NIHSS elements. Interrater reliability was near perfect (r2=0.98, P<0.001). Agreement between prospective and retrospective NIHSS score was also excellent (r2=0.94, P<0.001) and there was no systematic bias in retrospective scores. Agreement for individual items was moderate to high for all items except level of consciousness.
ConclusionsRetrospective NIHSS scoring with the algorithm is reliable and unbiased even when physical examination elements are missing from the written record. Stroke research using retrospective review of charts or of administrative databases should adjust for differences in stroke severity using such an algorithm.
Key Words: outcome prognosis stroke assessment
| Introduction |
|---|
|
|
|---|
When patient-level stroke outcomes are assessed, initial stroke severity is one of the variables that must be taken into account in adjusting outcomes for differences in case mix.1 2 This adjustment is critical, because it is well established that stroke severity at onset influences many outcomes, including mortality, length of stay, progression of deficit, and eventual functional recovery.3 4 5 6 7 8 Although initial stroke severity can be validly assessed in prospective studies, requisite variables are not typically recorded in a quantifiable way in most patient-care settings.
The NIH Stroke Scale (NIHSS) is a well validated and commonly used stroke impairment scale that sums the scores from individual elements of the neurological examination to provide an overall stroke impairment score.9 10 Although the NIHSS was recently reported to be reliable and valid for retrospective scoring,11 the study was conducted exclusively in stroke clinical trial participants, for whom the level of chart documentation is likely higher than in usual clinical practice. This report also did not address the reliability of individual NIHSS items. The aim of this research was to assess the validity and reliability of an algorithm for retrospective NIHSS scoring that would not require research-level documentation of the neurological examination. We hypothesized that (1) chart documentation for all elements of the NIHSS in nonclinical trial participants would be less than previously reported and (2) the algorithm devised for retrospective scoring would provide a reasonable estimate of initial stroke severity.
| Subjects and Methods |
|---|
|
|
|---|
Two raters certified in prospective administration of the NIHSS used the written algorithm to independently complete retrospective NIHSS assessments from photocopied admission H&Ps. If needed, H&Ps were edited before retrospective review to remove any mention of the NIHSS. Stroke Team notes, if present, were not used for retrospective scoring. One rater also noted which NIHSS elements were missing from the written H&P.
Linear regression was used to assess interrater reliability and to examine the relationship between retrospective (NIH-R) and prospective (NIH-P) scores. Because NIHSS scores are ordinal and not usually normally distributed, the Wilcoxon signed rank test was used to assess systematic differences in NIH-P and NIH-R. Weighted kappa statistics were calculated to determine the level of agreement of individual items of the NIH-P and NIH-R. Weighted kappa was used because NIHSS items are ordinal, not binary, and this technique adjusts for the amount of agreement expected by chance and the magnitude of individual disagreements. We used the usual quadratic weighting scheme, which bases disagreement weights on the square of the amount of discrepancy, thus making the weighted kappa equivalent to the intraclass correlation coefficient.12
| Results |
|---|
|
|
|---|
|
Interrater reliability between retrospective raters was nearly perfect
(Figure 1
; r=.99,
r2=0.98, P<0.001).
Agreement was also nearly perfect for overall prospective and
retrospective NIHSS score (Figure 2
;
r=.97, r2=0.94, ß=1.04,
95% CI ß=0.94, 1.14, P<0.001). NIH-R scores were not
significantly different from NIH-P scores (mean 5.3 and 5.4,
respectively; median 3 and 3, respectively; P=0.55 by
Wilcoxon signed rank test), which indicated no systematic bias
in retrospective scoring. Weighted kappa for agreement between
individual NIH-P and NIH-R items was moderate to high for all items
except level of consciousness, ranging from 0.54 for "commands" to
0.94 for "best gaze" (Table 2
). In
general, agreement was best among motor items and worst among
higher cortical function items.
|
|
|
| Discussion |
|---|
|
|
|---|
Kasner and colleagues11 also found that retrospective NIH scores could be reliably generated, with an 86% probability of correctly ranking NIH scores in 5-point interval categories at both admission and discharge. They also found that raters with varying levels of clinical stroke expertise (although all were certified in NIHSS prospective assessment, as in our study) were equally proficient at retrospective scoring. A major difference between our study and that of Kasner et al was that all but 1 of their 39 admission notes had "complete or near-complete" neurological examinations, while only 1 of our 32 notes was complete for all NIHSS items. The most likely reason for this difference is that, as expected, documentation about participants in clinical trials is more complete than documentation for nontrial participants. Our study thus extends the generalizability of retrospective NIHSS scoring, showing that our algorithm reliably estimates the initial NIHSS score in ischemic stroke patients, even in settings with incomplete documentation of the neurological examination.
Our data also provide an assessment of which of the NIHSS items are most accurate in retrospective assessment. We found that, in general, motor items such as limb motor function and gaze (eye movements) had higher agreement than cortical items such as level of consciousness or response to questions. One reason for this difference may be that motor response is usually assessed with a standardized scale, often the Medical Research Council scale, whereas higher cortical functions are recorded in a more individualized manner, thus potentially increasing the variability between observers. This result, however, is opposite to that reported in the retrospective validation of the Canadian Neurological Scale (CNS),13 in which Goldstein and colleagues found a higher level of agreement on speech, level of consciousness, and orientation items compared with motor items. This disparity may be related to the differing requirements for CNS and NIHSS items. For example, NIHSS items for commands and orientation require response to specific questions or commands and thus may be more difficult to accurately score retrospectively, whereas CNS motor items require both proximal and distal ratings for each limb and so may be less accurate when only a single motor score is recorded for each limb.
The importance of adjusting for initial stroke severity when assessing subsequent clinical outcomes cannot be overstated. Numerous studies have found that initial stroke severity is a strong predictor of many important outcomes, including mortality, length of stay, discharge destination, and functional outcomes.3 4 5 6 7 8 Most administrative databases include variables such as mortality, length of stay, and discharge destination as important clinical outcomes but do not include a measure of initial stroke severity. If such databases are used for comparing quality of stroke care between institutions or providers, adjustment must be made for case mix, including stroke severity. Our data provide an algorithm with which to reliably assess initial NIHSS scores for this purpose. Ultimately, our ability to demonstrate what constitutes high-quality stroke care will require us to link various structure and process variables to improved patient-level outcomes; this linkage is most convincingly demonstrated if databases prospectively record and appropriately adjust for differences in initial stroke severity.
Although we demonstrated excellent reliability for retrospective NIHSS scoring and for interrater NIHSS assessments, it is important to note that all raters were certified in prospective NIHSS assessment and thus may be more adept at interpreting written H&Ps in the context of the NIHSS examination. We also performed this study exclusively in teaching hospitals, where one could argue that the level of documentation of the neurological examination might be higher than in nonteaching settings. The effect that a higher proportion of missing examination items or that raters not trained in prospective NIHSS administration may have on the reliability of the retrospective algorithm is not known. We also had relatively few comatose patients, so the performance of the algorithm in these most-severe stroke patients remains to be determined.
Nonetheless, these data support the use of this retrospective algorithm to reliably score the NIHSS from written history and physical examinations, even when specific NIHSS items are missing from the written record. The use of such an algorithm will enhance our understanding of the influence of initial stroke severity on subsequent outcomes and will improve our ability to draw meaningful conclusions about stroke outcomes from administrative databases.
|
|
| Acknowledgments |
|---|
| Footnotes |
|---|
Received December 3, 1999; revision received January 12, 2000; accepted January 12, 2000.
| References |
|---|
|
|
|---|
2. Irwin P, Rudd A, for the Intercollegiate Working Party for Stroke. Casemix and process indicators of outcome in stroke: the Royal College of Physicians minimum data set for stroke. J R Coll Physicians Lond. 1998;32:442444.[Medline] [Order article via Infotrieve]
3. Oxbury JM, Breenhall RCD, Grainger KMR. Predicting the outcome of stroke: acute stage after cerebral infarction. BMJ. 1975;3:125127.
4.
Heinemann AW, Roth EJ, Cichowski K, Betts HB.
Multivariate analysis of improvement and
outcome following stroke rehabilitation. Arch Neurol. 1989;44:11671172.
5.
Muir KW, Weir CJ, Murray GD, Povey C, Lees KR.
Comparison of neurological scales and scoring systems for acute stroke
prognosis. Stroke. 1996;27:18171820.
6. Westling B, Norrving B, Thorngren M. Survival following stroke: a prospective population-based study of 438 hospitalized cases with prediction according to subtype, severity and age. Acta Neurol Scand. 1990;81:457463.[Medline] [Order article via Infotrieve]
7.
Samuelsson M, Soderfeldt B, Olsson GB. Functional
outcome in patients with lacunar infarction. Stroke. 1996;27:842846.
8.
DeGraba TJ, Hallenbeck JM, Pettigrew KD, Dutka AJ,
Kelly BJ. Progression in acute stroke: value of the initial NIH Stroke
Scale score on patient stratification in future trials.
Stroke. 1999;30:12081212.
9.
Brott T, Adams HP Jr, Olinger CP, Marler JR, Barsan
WG, Biller J, Spilker J, Holleran R, Eberle R, Herztberg V, Rorick M,
Moonaw CJ, Walker M. Measurement of acute cerebral infarction: a
clinical examination scale. Stroke. 1989;20:864870.
10.
Goldstein LB, Bertels C, Davis JN. Interrater
reliability of the NIH Stroke Scale. Arch Neurol. 1989;46:660662.
11.
Kasner SE, Chalela JA, Luciano JM, Cucchiara BL, Raps
EC, McGarvey ML, Conroy MB, Localio AR. Reliability and validity of
estimating the NIH Stroke Scale score from medical records.
Stroke. 1999;30:15341537.
12. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. 1973;33:613619.
13.
Goldstein LB, Chilukuri V. Retrospective
assessment of initial stroke severity with the Canadian Neurological
Scale. Stroke. 1997;28:11811184.
This article has been cited by other articles:
![]() |
J. Putaala, S. Curtze, S. Hiltunen, H. Tolppanen, M. Kaste, and T. Tatlisumak Causes of Death and Predictors of 5-Year Mortality in Young Adults After First-Ever Ischemic Stroke: The Helsinki Young Stroke Registry Stroke, August 1, 2009; 40(8): 2698 - 2703. [Abstract] [Full Text] [PDF] |
||||
![]() |
F. R. Pezzella, O. Picconi, A. De Luca, P. D. Lyden, and M. Fiorelli Development of the Italian Version of the National Institutes of Health Stroke Scale: It-NIHSS Stroke, July 1, 2009; 40(7): 2557 - 2559. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. Putaala, T. M. Metso, A. J. Metso, E. Makela, E. Haapaniemi, O. Salonen, M. Kaste, and T. Tatlisumak Thrombolysis in Young Adults With Ischemic Stroke Stroke, June 1, 2009; 40(6): 2085 - 2091. [Abstract] [Full Text] [PDF] |
||||
![]() |
B. Kissela, C. J. Lindsell, D. Kleindorfer, K. Alwell, C. J. Moomaw, D. Woo, M. L. Flaherty, E. Air, J. Broderick, and J. Tsevat Clinical Prediction of Functional Outcome After Ischemic Stroke: The Surprising Importance of Periventricular White Matter Disease and Race Stroke, February 1, 2009; 40(2): 530 - 536. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Gonzalez-Fernandez, J. T. Kleinman, P. K.S. Ky, J. B. Palmer, and A. E. Hillis Supratentorial Regions of Acute Ischemia Associated With Clinically Important Swallowing Disorders: A Pilot Study Stroke, November 1, 2008; 39(11): 3022 - 3028. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. F. Gottesman, J. T. Kleinman, C. Davis, J. Heidler-Gary, M. Newhart, V. Kannan, and A. E. Hillis Unilateral neglect is more severe and common in older patients with right hemispheric stroke Neurology, October 28, 2008; 71(18): 1439 - 1444. [Abstract] [Full Text] [PDF] |
||||
![]() |
A. E. Hillis, L. Gold, V. Kannan, L. Cloutman, J. T. Kleinman, M. Newhart, J. Heidler-Gary, C. Davis, E. Aldrich, R. Llinas, et al. Site of the ischemic penumbra as a predictor of potential for recovery of functions Neurology, July 15, 2008; 71(3): 184 - 189. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. Juhl Majersik, M. A. Smith, D. B. Zahuranec, B. N. Sanchez, and L. B. Morgenstern Population-Based Analysis of the Impact of Expanding the Time Window for Acute Stroke Treatment Stroke, December 1, 2007; 38(12): 3213 - 3217. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. S. Williams, T. Bakas, E. Brizendine, L. Plue, W. Tu, H. Hendrie, and K. Kroenke How Valid Are Family Proxy Assessments of Stroke Patients' Health-Related Quality of Life? Stroke, August 1, 2006; 37(8): 2081 - 2085. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. G. Stead, R. M. Gilmore, K. C. Vedula, A. L. Weaver, W. W. Decker, and R. D. Brown Jr Impact of acute blood pressure variability on ischemic stroke outcome Neurology, June 27, 2006; 66(12): 1878 - 1881. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. G. Stead, R. M. Gilmore, W. W. Decker, A. L. Weaver, and R. D. Brown Jr Initial emergency department blood pressure as predictor of survival after acute ischemic stroke Neurology, October 25, 2005; 65(8): 1179 - 1183. [Abstract] [Full Text] [PDF] |
||||
![]() |
U. Fischer, M. Arnold, K. Nedeltchev, C. Brekenfeld, P. Ballinari, L. Remonda, G. Schroth, and H. P. Mattle NIHSS Score and Arteriographic Findings in Acute Ischemic Stroke Stroke, October 1, 2005; 36(10): 2121 - 2125. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. W. Duncan, R. Zorowitz, B. Bates, J. Y. Choi, J. J. Glasberg, G. D. Graham, R. C. Katz, K. Lamberty, and D. Reker Management of Adult Stroke Rehabilitation Care: A Clinical Practice Guideline Stroke, September 1, 2005; 36(9): e100 - e143. [Full Text] [PDF] |
||||
![]() |
A. I. Qureshi, J. F. Kirmani, M. A. Sayed, A. Safdar, S. Ahmed, R. Ferguson, L. A. Hershey, K. J. Qazi, and for the Buffalo Metropolitan Area and Erie County Time to hospital arrival, use of thrombolytics, and in-hospital outcomes in ischemic stroke Neurology, June 28, 2005; 64(12): 2115 - 2120. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. K. Schiemanck, M. W.M. Post, Th. D. Witkamp, L. J. Kappelle, and A. J.H. Prevo Relationship between Ischemic Lesion Volume and Functional Status in the 2nd Week after Middle Cerebral Artery Stroke Neurorehabil Neural Repair, June 1, 2005; 19(2): 133 - 138. [Abstract] [PDF] |
||||
![]() |
L. S. Williams, E. J. Brizendine, L. Plue, T. Bakas, W. Tu, H. Hendrie, and K. Kroenke Performance of the PHQ-9 as a Screening Tool for Depression After Stroke Stroke, March 1, 2005; 36(3): 635 - 638. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. L. Brown, L. D. Lisabeth, N. M. Garcia, M. A. Smith, and L. B. Morgenstern Emergency department evaluation of ischemic stroke and TIA: The BASIC Project Neurology, December 28, 2004; 63(12): 2250 - 2254. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. B. Morgenstern, M. A. Smith, L. D. Lisabeth, J. M. H. Risser, K. Uchino, N. Garcia, P. J. Longwell, D. A. McFarling, O. Akuwumi, A. Al-Wabil, et al. Excess Stroke in Mexican Americans Compared with Non-Hispanic Whites: The Brain Attack Surveillance in Corpus Christi Project Am. J. Epidemiol., August 15, 2004; 160(4): 376 - 383. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. M. Cestari, D. M. Weine, K. S. Panageas, A. Z. Segal, and L. M. DeAngelis Stroke in patients with cancer: Incidence and etiology Neurology, June 8, 2004; 62(11): 2025 - 2030. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. S. Williams, S. S. Ghose, and R. W. Swindle Depression and Other Mental Health Diagnoses Increase Mortality Risk After Ischemic Stroke Am J Psychiatry, June 1, 2004; 161(6): 1090 - 1095. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. B. Morgenstern, L. D. Lisabeth, A. C. Mecozzi, M. A. Smith, P. J. Longwell, D. A. McFarling, and J. M.H. Risser A population-based study of acute stroke and TIA diagnosis Neurology, March 23, 2004; 62(6): 895 - 900. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. G. Merino and K. M. Heilman Editorial Comment--Measurement of Cognitive Deficits in Acute Stroke Stroke, October 1, 2003; 34(10): 2396 - 2398. [Full Text] [PDF] |
||||
![]() |
N U Weir, C E Counsell, M McDowall, A Gunkel, and M S Dennis Reliability of the variables in a new set of models that predict outcome after stroke J. Neurol. Neurosurg. Psychiatry, April 1, 2003; 74(4): 447 - 451. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. E. Kasner, B. L. Cucchiara, M. L. McGarvey, J. M. Luciano, D. S. Liebeskind, and J. A. Chalela Modified National Institutes of Health Stroke Scale Can Be Estimated From Medical Records Stroke, February 1, 2003; 34(2): 568 - 570. [Abstract] [Full Text] [PDF] |
||||
![]() |
A. Semplicini, A. Maresca, G. Boscolo, M. Sartori, R. Rocchi, V. Giantin, P. L. Forte, and A. C. Pessina Hypertension in Acute Ischemic Stroke: A Compensatory Mechanism or an Additional Damaging Factor? Arch Intern Med, January 27, 2003; 163(2): 211 - 216. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Schlegel;, S. J. Kolb, J. M. Luciano, J. M. Tovar, B. L. Cucchiara, D. S. Liebeskind, and S. E. Kasner Utility of the NIH Stroke Scale as a Predictor of Hospital Disposition Stroke, January 1, 2003; 34(1): 134 - 137. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. Appelros, I. Nydevik, A. Seiger, and A. Terent Predictors of Severe Stroke: Influence of Preexisting Dementia and Cardiac Disorders Stroke, October 1, 2002; 33(10): 2357 - 2362. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. M. Bravata, N. Kim, J. Concato, H. M. Krumholz, and L. M. Brass Thrombolysis for Acute Stroke in Routine Clinical Practice Arch Intern Med, September 23, 2002; 162(17): 1994 - 2001. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. S. Williams, J. Rotich, R. Qi, N. Fineberg, A. Espay, A. Bruno, S. E. Fineberg, and W. R. Tierney Effects of admission hyperglycemia on mortality and costs in acute ischemic stroke Neurology, July 9, 2002; 59(1): 67 - 71. [Abstract] [Full Text] [PDF] |
||||
![]() |
B. C. Meyer, T. M. Hemmen, C. M. Jackson, and P. D. Lyden Modified National Institutes of Health Stroke Scale for Use in Stroke Clinical Trials: Prospective Reliability and Validity Stroke, May 1, 2002; 33(5): 1261 - 1266. [Abstract] [Full Text] [PDF] |
||||
![]() |
W.S. Burgin, L. Staub, W. Chan, PhD;, T.H. Wein, R.A. Felberg, J.C. Grotta, A.M. Demchuk, S.L. Hickenbottom, and L.B. Morgenstern Acute stroke care in non-urban emergency departments Neurology, December 11, 2001; 57(11): 2006 - 2012. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. C. Grotta, W. S. Burgin, A. El-Mitwalli, M. Long, M. Campbell, L. B. Morgenstern, M. Malkoff, and A. V. Alexandrov Intravenous Tissue-Type Plasminogen Activator Therapy for Ischemic Stroke: Houston Experience 1996 to 2000 Arch Neurol, December 1, 2001; 58(12): 2009 - 2013. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. E. Kasner, A. M. Demchuk, J. Berrouschot, E. Schmutzhard, L. Harms, P. Verro, J. A. Chalela, R. Abbur, H. McGrade, I. Christou, et al. Predictors of Fatal Brain Edema in Massive Hemispheric Ischemic Stroke Stroke, September 1, 2001; 32(9): 2117 - 2123. [Abstract] [Full Text] [PDF] |
||||
![]() |
C. D. Bushnell, D. C.C. Johnston, and L. B. Goldstein Retrospective Assessment of Initial Stroke Severity : Comparison of the NIH Stroke Scale and the Canadian Neurological Scale Stroke, March 1, 2001; 32(3): 656 - 660. [Abstract] [Full Text] [PDF] |
||||
![]() |
A. M. Lopez-Yunez, A. Bruno, L. S. Williams, E. Yilmaz, C. Zurru, and J. Biller Protocol Violations in Community-Based rTPA Stroke Treatment Are Associated With Symptomatic Intracerebral Hemorrhage Stroke, January 1, 2001; 32(1): 12 - 16. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Stroke Home | Subscriptions | Archives | Feedback | Authors | Help | AHA Journals Home | Search Copyright © 2000 American Heart Association, Inc. All rights reserved. Unauthorized use prohibited. |