Inter- and Intra-Rater Reliability of the Visual Vertical in Subacute Stroke
Background and Purpose—Visual vertical (VV) has been used increasingly as a routine clinical assessment to identify alteration of verticality perception as a possible cause of postural disorders after stroke. This study aims to determine whether the reliability of VV is sufficient to support a wide clinical use in neurorehabilitation for monitoring of patients with stroke.
Methods—Twenty patients with subacute stroke in neurorehabilitation unit were tested after a first and unique hemispheric stroke. To evaluate the inter-rater reliability, VV was assessed the same day by 2 examiners whose degrees of expertise differed. The second examiner repeated the test the next day to investigate intrarater reliability. VV orientation (mean, primary criterion) and uncertainty (SD, secondary criterion) were calculated for 10 trials. Their reliability was quantified by the intraclass correlation coefficient, Bland–Altman plots, and the minimal detectable change. The concordance between 2 examiners was quantified by Cohen’s κ coefficients (κ).
Results—About VV orientation, inter- and intrarater reliability were excellent (intraclass correlation coefficient, 0.979 and 0.982). The Bland–Altman plots and the minimal detectable change revealed a difference inferior to 2° between 2 tests. The concordance between 2 assessments for the diagnosis of abnormal VV orientation was absolute for the same examiner (κ=1; P<0.05) and excellent between 2 examiners (κ=0.92; P<0.05). As for VV uncertainty the intrarater reliability was satisfactory (intraclass correlation coefficient, 0.836) but the inter-rater reliability was poor (intraclass correlation coefficient, 0.211).
Conclusions—The orientation of the VV is a highly reliable criterion, which may be used both in research and in routine clinical practice.
In stroke rehabilitation, verticality perception has been progressively integrated into routine assessments1 as a basic spatial cognition measurement to complement other tests.2 Assessing verticality perception is considered clinically relevant to guide poststroke rehabilitation1 and monitor postural recovery3 because it informs the causes of the body’s misorientation with respect to vertical (lateropulsion). The visual vertical (VV) is the most commonly used test to assess verticality perception both in research and in clinical practice.3–10 It is a simple test that consists of adjusting a luminous rod to the vertical in darkness. VV perception is estimated by 2 measures: VV orientation,3–6,8–10 which reflects the direction perceived as vertical by individuals, and VV uncertainty,3,8,10,11 which reflects the robustness of the internal model of verticality.8,11
Although VV has been progressively generalized for poststroke assessment, with a bewildering array of paradigms,3–6,8–10 its psychometric properties remain to be established. Thus, the aim of this study was to determine whether VV measurements have enough reliability to support the use both in research and in routine clinical practice. We hypothesized that using a previously validated methodological approach for VV assessment,5,8 VV orientation and VV uncertainty would be reliable measures with higher intrarater than inter-rater reliability.
Materials and Methods
We assessed 20 consecutive patients with subacute stroke (64±15 years; 8 women, 12 men; 8 left- and 12 right-sided lesions) admitted in a neurorehabilitation unit for a single recent (5.3±2.6 weeks) hemorrhagic (7/20) or ischemic (13/20) hemispheric stroke. Demographic and clinical data gathered dealt with stroke features, functional outcome, postural behavior, spatial neglect, aphasia, and depression. Individual and mean data are shown in Supplement Data I in the online-only Data Supplement (Figure I in the online-only Data Supplement and Table I in the online-only Data Supplement). Patients were excluded if they had brain stem or cerebellar strokes involving vestibulo-ocular symptoms because in those patients a different mechanism is involved in abnormal VV perception.5–7 We also excluded patients with neuropathy, psychiatric disorders, or major comprehension problems because of aphasia or dementia and those with unstable medical problems. Participants gave their informed consent according to the guidelines of the institutions’ ethics committees.
Setup for VV Measurement
VV perception was assessed in darkness by 10 adjustments of the direction of a bright line (15 cm long, 2 mm wide) presented on a computer screen, at eye level, using a previously validated methodological approach for VV assessment.5,8 Patients were seated with their head and trunk maintained upright5,11 and were asked to verbally adjust the line to the vertical. There was no time limit or feedback from the examiner. The measures started after 2 minutes of darkness. For each patient and each session, we calculated 2 criteria for the 10 trials: the mean VV orientation as a primary criterion3–6,8–10 and the mean VV uncertainty (within-subject SD) as a secondary criterion.3,8,10,11 According to standards, after sign transformation according to the lesion side, a negative value corresponded to a contralesional tilt. Patients were categorized as having normal VV perception (from −2.5° to 2.5°), contralesional (<−2.5°), or ipsilesional (>2.5°) bias according to normal ranges validated in a previous study.5
VV was assessed 3 times. In the first session, 2 examiners (A and B), whose degrees of expertise differed (a novice versus an expert in VV assessment), performed independent successive measurements (A1 and B1) with a 5-minute resting period in between. During the resting period, patients kept their eyes open in a lit environment and could adopt a spontaneous sitting position. Each examiner had to install the patients before each measurement recording. The order of assessment by examiners A and B was counterbalanced between patients. The examiner was blinded to the patients’ VV estimates measured by the other examiner. In the second session, the next day at the same time, examiner B repeated the entire procedure (B2 measure).
The distribution of VV orientation and VV uncertainty tested using Kolmogorov–Smirnov test was normal allowing the use of parametric statistics.
Inter-rater reliability (examiners A1 versus B1) and intrarater reliability (same examiner B1 versus B2) were determined by calculating the intraclass correlation coefficient (ICC).12 Absolute reliability was assessed with the Bland–Altman method with 95% confidence intervals,13 and with the SE of measurement (SEM=SDX×√1−ICC).14 SEM was then used to estimate the minimal detectable change (MDC95=1.96×SEM×√2)14 for 95% degree of confidence on VV, the amount by which a patient’s score needs to change to ensure that the change is greater than measurement error (see details in Supplementary Data II).
In addition, we used Cohen’s κ coefficient (κ) to calculate the percentage agreement between the 2 examiners for the classification of VV perception for patients (contralesional bias, ipsilesional bias, no bias) in regards to normative data.5 A κ or ICC was considered in term of agreement: under 0.6, questionable; 0.6 to 0.79, moderate; and >0.8, high.15
Statistical tests were performed with SPSS 21 (IBM Corporation, Armonk, NY). The α risk was set at P≤0.05. Unless other indication, data are presented in the form mean±SD.
The average VV orientation was −3° and the average VV uncertainty was 1.6° (first assessment regardless of examiner). VV perception was normal for 8 patients (40%) and was abnormally tilted to the contralesional side for 9 (45%; −7.2±2.8°) and toward the ipsilesional side for 3 (15%; 2.8±0.1°). Individual data for each examiner and each measurement are shown in the Table (Supplementary Data I in the online-only Data Supplement).
The inter- rater reliability was excellent, with an ICC of 0.979 (Table; Figure 1A). The Bland–Altman plots (Figure 1B) showed that all of the differences but one were within the 95% limits of agreement (−2.3° to 2.6°). The mean difference between A1 and B1 measurements was small (0.1±1.2°), meaning that differences between the 2 examiners were acceptable for VV orientation. The MDC95 indicated that a change of ≥1.85° was required to be 95% certain that the change was not because of measurement error.
The agreement was excellent between the 2 examiners in discriminating normal from abnormal VV perception (κ=0.89; P<0.05) and more precisely in discriminating contralesional, ipsilesional, and no VV bias (κ=0.92; P<0.05).
The intrarater reliability was excellent, with an ICC of 0.982 (Figure 1C). The Bland–Altman plots (Figure 1D) showed that all the differences were within the 95% limits of agreement (−2° to 1.9°) with a low mean difference between B1 and B2 measurements (−0.05±1°), meaning that when retested, the VV orientation measure was similar. The MDC95 was 1.74°.
The agreement between the 2 measurements from a same examiner (B1 and B2) was absolute in discriminating normal from abnormal VV perception (κ=1; P<0.05) and more precisely in discriminating contralesional, ipsilesional, and no VV bias (κ=1; P<0.05).
Although the inter-rater reliability (Table; Figure 2A) was poor as determined by an ICC of 0.21, the Bland–Altman plots (Figure 2B) showed that all of the differences but one were within the 95% limits of agreement (−2.6° to 1.6°). The mean difference between A1 and B1 uncertainty measures was small (−0.5±1.08°) meaning that differences between 2 examiners were clinically acceptable for VV uncertainty. The MDC95 was 2.62°.
Intrarater reliability (Figure 2C) was excellent (ICC=0.836). The Bland–Altman plots (Figure 2D) showed that most differences were within the 95% limits of agreement (−1.03° to 1.33°) with a small mean difference between B1 and B2 uncertainty measures (0.15±0.6°). This meant that when retested, the VV uncertainty was similar. The MDC95 was 1.54°.
The clinical characteristics of our patients and the frequency and magnitude of VV perception were representative of those previously reported in the literature for a large cohort of patients with stroke.5,6,10
Our study showed that VV orientation is a reliable measure for patients with subacute stroke and has excellent inter- and intrarater reliability (ICCs >0.98) and acceptable measurement error (<0.7°). We found an MDC <2°, which means that a change ≥2° can be interpreted with 95% confidence as a real change. This threshold is critical for clinical rehabilitation settings in the follow-up of patients with stroke with a tilted perception of verticality but also for clinical stroke trials to analyze the responsiveness to a specific rehabilitation program.
For VV uncertainty, our results revealed a poor relative reliability about ICC values, especially for inter-rater reliability. Yet, absolute reliability (Bland–Altman analysis13 and SEM14) remained clinically satisfactory. In the present study, the 2 examiners had different degrees of expertise, which might explain their different VV uncertainties and affect the reliability of this measure. This suggests that clear guidelines for patients’ installation, instructions, and handling, as well as training would benefit examiners. Lastly, the lowest ICCs for the VV uncertainty is also affected by the lower intersubject variance (to 0.6° from 3.9°) as compared with VV orientation (to −12.8° from 4.2°). ICC values are the ratio of the variance between subjects and the total variance and thus coefficients are systematically lower in homogeneous samples.14 This also explained why the outsider had more influence on VV uncertainty than VV orientation.
These findings are valid for the conditions under which VV was assessed in our study, that is, patients with subacute hemispheric stroke seated with trunk and head maintained upright. Moreover, in our study, patients verbally adjusted the visual line to the vertical. In some studies, patients estimated VV by manipulating a remote control.6,7 Similar to others3,4 we preferred using the verbal modality to circumvent possible biases because of handedness, apraxia, learning, or executive dysfunctions. The verbal modality does not preclude testing aphasic patients, because a code answer can easily be set up for these patients.
Our study reveals that inter- and intrarater reliability of the VV orientation is excellent as a measure of verticality perception in patients with subacute hemispheric stroke.
We thank the neuropsychologist student and clinicians of the neurorehabilitation unit of the University Hospital of Grenoble.
Sources of Funding
This work has been supported by a grant from La Société Francophone Posture Equilibre Locomotion.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.115.009610/-/DC1.
- Received April 8, 2015.
- Revision received April 8, 2015.
- Accepted April 14, 2015.
- © 2015 American Heart Association, Inc.
- Bonan IV,
- Hubeaux K,
- Gellez-Leman MC,
- Guichard JP,
- Vicaut E,
- Yelnik AP.
- Pérennou DA,
- Mazibrada G,
- Chauvineau V,
- Greenwood R,
- Rothwell J,
- Gresty MA,
- et al
- Barra J,
- Marquer A,
- Joassin R,
- Reymond C,
- Metge L,
- Chauvineau V,
- et al
- Saj A,
- Honoré J,
- Bernati T,
- Coello Y,
- Rousseaux M.
- Yelnik AP,
- Lebreton FO,
- Bonan IV,
- Colle FM,
- Meurin FA,
- Guichard JP,
- et al