(Stroke. 1997;28:307-310.)
© 1997 American Heart Association, Inc.
Articles |
the Divisions of Neurology (L.B.G.) and General Internal Medicine (G.P.S.), Department of Medicine, the Department of Community and Family Medicine (Biometry) (G.P.S.), and the Center for Health Policy Research and Education (L.B.G., G.P.S.), Duke University; and the Division of Neurology (L.B.G.), Durham Department of Veterans Affairs Medical Center, Durham, NC.
Correspondence to Larry B. Goldstein, MD, Box 3651, Duke University Medical Center, Durham, NC 27710. E-mail golds004@mc.duke.edu.
| Abstract |
|---|
|
|
|---|
Methods In anticipation of the initiation of a randomized trial of a new therapy for patients with acute ischemic stroke, 30 physician investigators (30% of whom were not neurologists) and 29 non-physician study coordinators were trained in the use of the NIHSS at an informational and training conference using standardized videotaped patient examinations. A series of 4 patients were rated initially. After 3 months, the same 4 patients were rerated, providing a measure of intraobserver reliability. An additional series of 4 new patients were also rated after 3 months and, with the initial 4 ratings, provided data for assessment of interobserver reliability.
Results Overall, 28% of the raters had previous experience with the NIHSS, and 22% had previously used the videotapes as used in the present trial. The coefficients of determination (r2) were each greater than .95 when the means of the two ratings of the same 4 cases were compared between (1) neurologists and other types of physicians, (2) physicians and study coordinators, (3) raters who had prior experience with the NIHSS and those without prior experience, and (4) raters who had used the videotapes in the past and those who had never viewed the tapes. The calculated r2s were greater than .98 for the initial rating of the first 4 cases and for the later rating of the 4 new cases. The slopes of the regression lines were all near 1, indicating that the raters were similarly calibrated. The intraclass correlation coefficients were .93 and .95, reflecting high levels of intraobserver and interobserver reliability.
Conclusions These data extend the previously demonstrated reliability of the NIHSS to non-neurologists and show that both a variety of physician investigators and nurse study coordinators can be rapidly trained to reliably apply the scale in the context of an actual clinical trial.
Key Words: clinical trials stroke assessment
| Introduction |
|---|
|
|
|---|
| Subjects and Methods |
|---|
|
|
|---|
|
After this instructional session, the participants were asked to independently rate a series of 4 videotaped stroke patients with a wide range of impairments (Henry Ford Hospital and Health Sciences Center NIHSS Training Tape Cases 1 through 4)3 using the NIHSS to provide an initial measure of interobserver reliability. Data were also collected concerning physician specialty, study coordinator training, whether the rater had used the NIHSS in the past, and whether the rater had used the NIHSS training and certification videotapes in the past. This session was followed by a discussion period during which any uncertainties in the use of the NIHSS were addressed. After 3 months, the investigators and coordinators were asked to rerate the 4 videotaped stroke patients that they had scored at the earlier training session without reference to their prior responses to provide a measure of intraobserver reliability. In addition, they were asked to rate 4 new cases (Henry Ford Hospital and Health Sciences Center NIHSS Certification Tape 1, Cases 1 through 4) to both provide another measure of interobserver reliability and determine whether reliability had diminished since the initial training.
Mean scores were first calculated for each case on the basis of the various groupings. The degree to which the scores were calibrated between groups of raters was then determined with linear regression analyses. Overall levels of agreement were assessed with intraclass correlation coefficients (ICC).7 After ANOVA indicated that the variance components for physician specialty, role (physician versus study coordinator), previous NIHSS experience, previous NIHSS videotape experience, and time were not statistically distinguishable from 0, the ICCs were calculated as ICC=
s2/(
s2+
e2), where
s2 is the variance component for subject and
e2 is the variance component for the residual error. The ICC will be high if raters are similarly calibrated (ie, the slopes of the regression lines are near 1) and the variation between subjects is large relative to the variation between raters. A calculated ICC=1 reflects perfect reliability. Frequencies were compared with
2 statistics.
| Results |
|---|
|
|
|---|
2=0.041, P=.84) or the videotapes (23% of physicians versus 21% of study coordinators;
2=0.060, P=.81).
Fig 1
gives plots of the mean scores for neurologists compared with other types of physicians, for physicians compared with study coordinators, for raters who had prior experience with the NIHSS compared with those without prior experience, and for raters who had used the videotapes in the past compared with those who had never viewed the videotapes for both the 4 cases rated during the initial training session and for the second group of 4 new cases viewed 3 months later. The mean scores reflect patients with both mild and severe impairments. The coefficients of determination (r2) are greater than .98 for each comparison, indicating that the scores given by each group account for more than 98% of the variance in the scores given by the respective comparison group. The slopes of the regression lines are all near 1, indicating that the raters were similarly calibrated in their scorings (ie, a lack of systematic differences in the use of the scale). Fig 2
gives plots of the mean scores for the same groups for the 4 cases rated during the initial training session and the repeated scores of the same 4 cases 3 months later. The calculated r2s are all greater than .95, and the slopes of the regression lines are all near 1.
|
|
The ICC was .94 for the 4 cases rated at the initial training session and .92 for the 4 new cases rated 3 months later. The overall ICC based on the ratings of these 8 cases was .95, reflecting a high level of interobserver reliability. The ICC was .93 for the cases rated during the initial training session and rerated after 3 months had elapsed, indicating a high level of intraobserver reliability.
| Discussion |
|---|
|
|
|---|
The reliabilities of the individual items that compose the NIHSS have been studied extensively, both with the videotapes used in the present study and with live patients.3 4 5 6 The items rating facial paresis and limb ataxia were consistently found to be the least reliable. The scoring of these items and other items that have proved difficult in certain types of patients was stressed to the raters. Although there was some decrease in reliability between the initial scoring of the training-videotape patients (ICC=.94) and the scoring of the certification-videotape patients 3 months later (ICC=.92), the difference was small and not statistically significant. However, this decrease in reliability could have increased in magnitude with time, the so-called "drift effect."4 Because the present trial was not initiated, we do not have further longitudinal data to address this issue. Reliability assessments and periodic recertification during the course of a long clinical trial may be necessary.
| Acknowledgments |
|---|
Received June 27, 1996; revision received August 23, 1996; accepted August 23, 1996.
| References |
|---|
|
|
|---|
2. Platz T, Denzler P, Kaden B, Mauritz K-H. Motor learning after recovery from hemiparesis. Neuropsychologia. 1994;32:1209-1223.[Medline] [Order article via Infotrieve]
3. Lyden P, Brott T, Tilley B, Welch KMA, Mascha EJ, Levine S, Haley EC, Grotta J, Marler J, for the NINDS TPA Stroke Study Group. Improved reliability of the NIH Stroke Scale using video training. Stroke. 1994;25:2220-2226.[Abstract]
4. Albanese MA, Clarke WR, Adams HP Jr, Woolson RF, for the TOAST Investigators. Ensuring reliability of outcome measures in multicenter clinical trials of treatments for acute ischemic stroke: the program developed for the Trial of ORG 10172 in Acute Stroke Treatment (TOAST). Stroke. 1994;25:1746-1751.[Abstract]
5.
Brott T, Adams HP Jr, Olinger CP, Marler JR, Barsan WG, Biller J, Spilker J, Holleran R, Eberle R, Hertzberg V, Rorick M, Moomaw CJ, Walker M. Measurements of acute cerebral infarction: a clinical examination scale. Stroke. 1989;20:864-870.
6.
Goldstein LB, Bertels C, Davis JN. Interrater reliability of the NIH stroke scale. Arch Neurol. 1989;46:660-662.
7. Fleiss JL. The Design and Analysis of Clinical Experiments. New York, NY: John Wiley & Sons Inc; 1986:1-31.
This article has been cited by other articles:
![]() |
L. B. Goldstein, P. Amarenco, J. Zivin, M. Messig, I. Altafullah, A. Callahan, M. Hennerici, M. J. MacLeod, H. Sillesen, R. Zweifler, et al. Statin Treatment and Stroke Outcome in the Stroke Prevention by Aggressive Reduction in Cholesterol Levels (SPARCL) Trial Stroke, November 1, 2009; 40(11): 3526 - 3531. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. H. Schwamm, R. G. Holloway, P. Amarenco, H. J. Audebert, T. Bakas, N. R. Chumbler, R. Handschu, E. C. Jauch, W. A. Knight IV, S. R. Levine, et al. A Review of the Evidence for the Use of Telemedicine Within Stroke Systems of Care: A Scientific Statement From the American Heart Association/American Stroke Association Stroke, July 1, 2009; 40(7): 2616 - 2634. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. Lyden, R. Raman, L. Liu, M. Emr, M. Warren, and J. Marler National Institutes of Health Stroke Scale Certification Is Reliable Across Multiple Venues Stroke, July 1, 2009; 40(7): 2507 - 2511. [Abstract] [Full Text] [PDF] |
||||
![]() |
F. R. Pezzella, O. Picconi, A. De Luca, P. D. Lyden, and M. Fiorelli Development of the Italian Version of the National Institutes of Health Stroke Scale: It-NIHSS Stroke, July 1, 2009; 40(7): 2557 - 2559. [Abstract] [Full Text] [PDF] |
||||
![]() |
W. A. Gray, S. Chaturvedi, P. Verta, and on behalf of the Investigators and the Executive C Thirty-Day Outcomes for Carotid Artery Stenting in 6320 Patients From 2 Prospective, Multicenter, High-Surgical-Risk Registries Circ Cardiovasc Interv, June 1, 2009; 2(3): 159 - 166. [Abstract] [Full Text] [PDF] |
||||
![]() |
L. D. Alexander, S. E. Black, K. K. Patterson, F. Gao, C. J. Danells, and W. E. McIlroy Association Between Gait Asymmetry and Brain Lesion Location in Stroke Patients Stroke, February 1, 2009; 40(2): 537 - 544. [Abstract] [Full Text] [PDF] |
||||
![]() |
D.C. Suh, J.K. Kim, J.W. Choi, B.S. Choi, H.W. Pyun, Y.J. Choi, M.-H. Kim, H.R. Yang, H.I. Ha, S.J. Kim, et al. Intracranial Stenting of Severe Symptomatic Intracranial Stenosis: Results of 100 Consecutive Patients AJNR Am. J. Neuroradiol., April 1, 2008; 29(4): 781 - 785. [Abstract] [Full Text] [PDF] |
||||
![]() |
H. P. Adams Jr, G. del Zoppo, M. J. Alberts, D. L. Bhatt, L. Brass, A. Furlan, R. L. Grubb, R. T. Higashida, E. C. Jauch, C. Kidwell, et al. Guidelines for the Early Management of Adults With Ischemic Stroke: A Guideline From the American Heart Association/American Stroke Association Stroke Council, Clinical Cardiology Council, Cardiovascular Radiology and Intervention Council, and the Atherosclerotic Peripheral Vascular Disease and Quality of Care Outcomes in Research Interdisciplinary Working Groups: The American Academy of Neurology affirms the value of this guideline as an educational tool for neurologists. Circulation, May 22, 2007; 115(20): e478 - e534. [Abstract] [Full Text] [PDF] |
||||
![]() |
R. Mikulik, M. Ribo, M. D. Hill, J. C. Grotta, M. Malkoff, C. Molina, M. Rubiera, R. Delgado-Mederos, J. Alvarez-Sabin, A. V. Alexandrov, et al. Accuracy of Serial National Institutes of Health Stroke Scale Scores to Identify Artery Status in Acute Ischemic Stroke Circulation, May 22, 2007; 115(20): 2660 - 2665. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. Roquer, A. Ois, A. Rodriguez-Campello, M. Gomis, E. Munteis, J. Jimenez-Conde, E. Cuadrado-Godia, and J. E. Martinez-Rodriguez Atherosclerotic Burden and Early Mortality in Acute Ischemic Stroke Arch Neurol, May 1, 2007; 64(5): 699 - 704. [Abstract] [Full Text] [PDF] |
||||
![]() |
H. P. Adams Jr, G. del Zoppo, M. J. Alberts, D. L. Bhatt, L. Brass, A. Furlan, R. L. Grubb, R. T. Higashida, E. C. Jauch, C. Kidwell, et al. Guidelines for the Early Management of Adults With Ischemic Stroke: A Guideline From the American Heart Association/ American Stroke Association Stroke Council, Clinical Cardiology Council, Cardiovascular Radiology and Intervention Council, and the Atherosclerotic Peripheral Vascular Disease and Quality of Care Outcomes in Research Interdisciplinary Working Groups: The American Academy of Neurology affirms the value of this guideline as an educational tool for neurologists Stroke, May 1, 2007; 38(5): 1655 - 1711. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. Mikdashi, B. Handwerger, P. Langenberg, M. Miller, and S. Kittner Baseline Disease Activity, Hyperlipidemia, and Hypertension Are Predictive Factors for Ischemic Stroke and Stroke Severity in Systemic Lupus Erythematosus Stroke, February 1, 2007; 38(2): 281 - 285. [Abstract] [Full Text] [PDF] |
||||
![]() |
M. Bondanelli, M. R. Ambrosio, A. Onofri, A. Bergonzoni, S. Lavezzi, M. C. Zatelli, D. Valle, N. Basaglia, and E. C. degli Uberti Predictive Value of Circulating Insulin-Like Growth Factor I Levels in Ischemic Stroke Outcome J. Clin. Endocrinol. Metab., October 1, 2006; 91(10): 3928 - 3934. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. Lyden, R. Raman, L. Liu, J. Grotta, J. Broderick, S. Olson, S. Shaw, J. Spilker, B. Meyer, M. Emr, et al. NIHSS Training and Certification Using a New Digital Video Disk Is Reliable Stroke, November 1, 2005; 36(11): 2446 - 2449. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. W. Duncan, R. Zorowitz, B. Bates, J. Y. Choi, J. J. Glasberg, G. D. Graham, R. C. Katz, K. Lamberty, and D. Reker Management of Adult Stroke Rehabilitation Care: A Clinical Practice Guideline Stroke, September 1, 2005; 36(9): e100 - e143. [Full Text] [PDF] |
||||
![]() |
P. D. Syme, A. W. Byrne, R. Chen, R. Devenny, and J. F. Forbes Community-Based Stroke Incidence in a Scottish Population: The Scottish Borders Stroke Study Stroke, September 1, 2005; 36(9): 1837 - 1843. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. K. Schiemanck, M. W.M. Post, Th. D. Witkamp, L. J. Kappelle, and A. J.H. Prevo Relationship between Ischemic Lesion Volume and Functional Status in the 2nd Week after Middle Cerebral Artery Stroke Neurorehabil Neural Repair, June 1, 2005; 19(2): 133 - 138. [Abstract] [PDF] |
||||
![]() |
L. B. Goldstein and D. L. Simel Is This Patient Having a Stroke? JAMA, May 18, 2005; 293(19): 2391 - 2402. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. H. Kim, C. L. Haney, and G. Van Ginhoven Utility of Outcome Measures After Treatment for Intracranial Aneurysms: A Prospective Trial Involving 520 Patients Stroke, April 1, 2005; 36(4): 792 - 796. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Gao, K. S. Wong, T. Hansberg, W. W. M. Lam, D. W. Droste, and E. B. Ringelstein Microembolic Signal Predicts Recurrent Cerebral Ischemic Events in Acute Stroke Patients With Middle Cerebral Artery Stenosis Stroke, December 1, 2004; 35(12): 2832 - 2836. [Abstract] [Full Text] [PDF] |
||||
![]() |
A. M. Nor, C. McAllister, S.J. Louw, A.G. Dyker, M. Davis, D. Jenkinson, and G.A. Ford Agreement Between Ambulance Paramedic- and Physician-Recorded Neurological Signs With Face Arm Speech Test (FAST) in Acute Stroke Patients Stroke, June 1, 2004; 35(6): 1355 - 1359. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. R. Lees Stroke Is Best Managed by a Neurologist: Battle of the Titans Stroke, November 1, 2003; 34(11): 2764 - 2765. [Full Text] [PDF] |
||||
![]() |
L. Perry and S. McLaren Implementing evidence-based guidelines for nutrition support in acute stroke Evid. Based Nurs., July 1, 2003; 6(3): 68 - 71. [Full Text] [PDF] |
||||
![]() |
M Uno, K T Kitazato, K Nishi, H Itabe, and S Nagahiro Raised plasma oxidised LDL in acute cerebral infarction J. Neurol. Neurosurg. Psychiatry, March 1, 2003; 74(3): 312 - 316. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. E. Kasner, B. L. Cucchiara, M. L. McGarvey, J. M. Luciano, D. S. Liebeskind, and J. A. Chalela Modified National Institutes of Health Stroke Scale Can Be Estimated From Medical Records Stroke, February 1, 2003; 34(2): 568 - 570. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Schlegel;, S. J. Kolb, J. M. Luciano, J. M. Tovar, B. L. Cucchiara, D. S. Liebeskind, and S. E. Kasner Utility of the NIH Stroke Scale as a Predictor of Hospital Disposition Stroke, January 1, 2003; 34(1): 134 - 137. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. L. Tirschwell, W.T. Longstreth Jr, K. J. Becker, R. E. Gammans Sr, L. A. Sabounjian, S. Hamilton, and L. B. Morgenstern Shortening the NIH Stroke Scale for Use in the Prehospital Setting Stroke, December 1, 2002; 33(12): 2801 - 2806. [Abstract] [Full Text] [PDF] |
||||
![]() |
B. C. Meyer, T. M. Hemmen, C. M. Jackson, and P. D. Lyden Modified National Institutes of Health Stroke Scale for Use in Stroke Clinical Trials: Prospective Reliability and Validity Stroke, May 1, 2002; 33(5): 1261 - 1266. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Kastenbauer, F. Winkler, H.-W. Pfister, J. F. Bruzzi, D. D. Brennan, Y. Tokuda, N. Nakazato, S. Segal, P. Tattevin, F. Bruneel, et al. Cranial CT before Lumbar Puncture in Suspected Meningitis N. Engl. J. Med., April 18, 2002; 346(16): 1248 - 1251. [Full Text] [PDF] |
||||
![]() |
R. Hasbun, J. Abrahams, J. Jekel, and V. J. Quagliarello Computed Tomography of the Head before Lumbar Puncture in Adults with Suspected Meningitis N. Engl. J. Med., December 13, 2001; 345(24): 1727 - 1733. [Abstract] [Full Text] [PDF] |
||||
![]() |
W.S. Burgin, L. Staub, W. Chan, PhD;, T.H. Wein, R.A. Felberg, J.C. Grotta, A.M. Demchuk, S.L. Hickenbottom, and L.B. Morgenstern Acute stroke care in non-urban emergency departments Neurology, December 11, 2001; 57(11): 2006 - 2012. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. Rundek, H. Mast, A. Hartmann, B. Boden-Albala, L. Lennihan, I.-F. Lin, M. C. Paik, and R. L. Sacco Predictors of resource use after acute hospitalization: The Northern Manhattan Stroke Study Neurology, October 24, 2000; 55(8): 1180 - 1187. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. W. Duncan, H. S. Jorgensen, and D. T. Wade Outcome Measures in Acute Stroke Trials : A Systematic Review and Some Recommendations to Improve Practice Stroke, June 1, 2000; 31(6): 1429 - 1438. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. M. Britt, J. E. Heiserman, R. M. Snider, H. A. Shill, C. R. Bird, and R. C. Wallace Incidence of Postangiographic Abnormalities Revealed by Diffusion-Weighted MR Imaging AJNR Am. J. Neuroradiol., January 1, 2000; 21(1): 55 - 59. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. P. Grichnik, A. J.J. Ijsselmuiden, T. A. D’Amico, D. H. Harpole Jr, W. D. White, J. A. Blumenthal, and M. F. Newman Cognitive decline after major noncardiac operations: a preliminary prospective study Ann. Thorac. Surg., November 1, 1999; 68(5): 1786 - 1791. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. Lyden, M. Lu, C. Jackson, J. Marler, R. Kothari, T. Brott, and J. Zivin Underlying Structure of the National Institutes of Health Stroke Scale : Results of a Factor Analysis Stroke, November 1, 1999; 30(11): 2347 - 2354. [Abstract] [Full Text] [PDF] |
||||
![]() |
A. G. Dyker, K. R. Edwards, P. B. Fayad, J. T. Hormes, and K. R. Lees Safety and Tolerability Study of Aptiganel Hydrochloride in Patients With an Acute Ischemic Stroke Stroke, October 1, 1999; 30(10): 2038 - 2042. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. E. Kasner, J. A. Chalela, J. M. Luciano, B. L. Cucchiara, E. C. Raps, M. L. McGarvey, M. B. Conroy, and A. R. Localio Reliability and Validity of Estimating the NIH Stroke Scale Score from Medical Records Stroke, August 1, 1999; 30(8): 1534 - 1537. [Abstract] [Full Text] [PDF] |
||||
![]() |
H. P. Adams Jr., P. H. Davis, E. C. Leira, K.-C. Chang, B. H. Bendixen, W. R. Clarke, R. F. Woolson, and M. D. Hansen Baseline NIH Stroke Scale score strongly predicts outcome after stroke: A report of the Trial of Org 10172 in Acute Stroke Treatment (TOAST) Neurology, July 1, 1999; 53(1): 126 - 126. [Abstract] [Full Text] |
||||
![]() |
A. G. Dyker and K. R. Lees Safety and Tolerability of GV150526 (a Glycine Site Antagonist at the N-Methyl-D-Aspartate Receptor) in Patients With Acute Stroke Stroke, May 1, 1999; 30(5): 986 - 992. [Abstract] [Full Text] [PDF] |
||||
![]() |
K.J. van Everdingen, J. van der Grond, L.J. Kappelle, L.M.P. Ramos, and W.P.T.M. Mali Diffusion-Weighted Magnetic Resonance Imaging in Acute Stroke Stroke, September 1, 1998; 29(9): 1783 - 1790. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. Schmulling, M. Grond, J. Rudolf, and P. Kiencke Training as a Prerequisite for Reliable Use of NIH Stroke Scale Stroke, June 1, 1998; 29(6): 1258 - 1259. [Full Text] |
||||
![]() |
L. Roberts and C. Counsell Assessment of Clinical Outcomes in Acute Stroke Trials Stroke, May 1, 1998; 29(5): 986 - 991. [Abstract] [Full Text] [PDF] |
||||
![]() |
H. P. Adams Jr Treating Ischemic Stroke as an Emergency Arch Neurol, April 1, 1998; 55(4): 457 - 461. [Abstract] [Full Text] [PDF] |
||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Stroke Home | Subscriptions | Archives | Feedback | Authors | Help | AHA Journals Home | Search Copyright © 1997 American Heart Association, Inc. All rights reserved. Unauthorized use prohibited. |