Color-Coded Fluid-Attenuated Inversion Recovery Images Improve Inter-Rater Reliability of Fluid-Attenuated Inversion Recovery Signal Changes Within Acute Diffusion-Weighted Image Lesions
Background and Purpose—Diffusion-weighted image fluid-attenuated inversion recovery (FLAIR) mismatch has been considered to represent ischemic lesion age. However, the inter-rater agreement of diffusion-weighted image FLAIR mismatch is low. We hypothesized that color-coded images would increase its inter-rater agreement.
Methods—Patients with ischemic stroke <24 hours of a clear onset were retrospectively studied. FLAIR signal change was rated as negative, subtle, or obvious on conventional and color-coded FLAIR images based on visual inspection. Inter-rater agreement was evaluated using κ and percent agreement. The predictive value of diffusion-weighted image FLAIR mismatch for identification of patients <4.5 hours of symptom onset was evaluated.
Results—One hundred and thirteen patients were enrolled. The inter-rater agreement of FLAIR signal change improved from 69.9% (k=0.538) with conventional images to 85.8% (k=0.754) with color-coded images (P=0.004). Discrepantly rated patients on conventional, but not on color-coded images, had a higher prevalence of cardioembolic stroke (P=0.02) and cortical infarction (P=0.04). The positive predictive value for patients <4.5 hours of onset was 85.3% and 71.9% with conventional and 95.7% and 82.1% with color-coded images, by each rater.
Conclusions—Color-coded FLAIR images increased the inter-rater agreement of diffusion-weighted image FLAIR recovery mismatch and may ultimately help identify unknown-onset stroke patients appropriate for thrombolysis.
Mismatch of lesion visibility between diffusion-weighted image and fluid-attenuated inversion recovery image (DWI-FLAIR mismatch) is a useful surrogate marker for estimating the lesion age in ischemic stroke.1–3 Reperfusion therapy based on multimodal imaging, including the information on DWI-FLAIR mismatch, is considered feasible and safe in patients with unclear-onset stroke.4
However, FLAIR signal intensity within an acute DWI lesion is heterogeneous, and subtle changes limited to the partial region were often regarded as negative for FLAIR signal change.5 Because the lesion conspicuity is subjective, inter-rater agreement of FLAIR signal change was low, especially in subtle changes.6 Quantitative analysis of FLAIR signal intensity ratio failed to increase the specificity for predicting ischemic lesions within the therapeutic window7 or the inter-rater agreement on FLAIR signal change.8
Color coding may increase the visual distinguishability, thus enabling clinicians to identify small changes in signal intensity. We hypothesized that color-coded FLAIR images would enhance the inter-rater agreement of DWI-FLAIR mismatch.
The records of patients with ischemic stroke admitted to Asan Medical Center <24 hours of clear onset between January and July 2013 were reviewed. Patients were excluded if (1) DWI lesion was absent, (2) images were obtained from other centers, or (3) imaging data quality was insufficient. Clinical data were obtained from a prospectively collected registry. The Institutional Review Board of the Asan Medical Center approved this study. Informed consent was not obtained because of the retrospective design.
Patients received a 1.5-T MRI scan (Magnetom Avanto; Siemens). The DWI slice with the largest lesion and the slice of corresponding FLAIR image were prepared. The FLAIR image slice was extracted as a DICOM file, converted to a NIfTI file and was presented on MRIcroN (http://www.nitrc.org/projects/mricron). The actc look-up table, which consists of 256 color codes, was used to assign the voxel color. The auto-contrast setting determined the intensity ranges (1–99 percentiles) of color codes. Finally, 3 corresponding slices of different sequences were prepared for each patient (Figure 1).
Measurement of Inter-Rater Agreement
Inter-rater agreement was measured in 2 pairs of raters blinded to clinical data (pair A: well-trained stroke neurologists; pair B: neurologists in residency training; see the online-only Data Supplement). To avoid a learning effect, rating was done with color-coded image first and a week later with the conventional image. FLAIR signal change was categorized into obvious (a hyperintense lesion within the DWI lesion), subtle (hyperintensity identified only with reference to the DWI lesion), or negative (no hyperintensity at all within the DWI lesion). Ten examples were provided as a reference before rating. FLAIR images rated negative were regarded as positive for DWI-FLAIR mismatch.
κ Value and percent agreement were used to quantify inter-rater agreement of FLAIR signal change and McNemar test to compare the difference. Patients with discrepant ratings by conventional images but concordant with color-coded images were regarded as patients with benefit from color-coded FLAIR image. Characteristics of these patients were compared with those of other patients using χ2 and t tests, appropriately. The predictive value of DWI-FLAIR mismatch for identification of patients <4.5 hours of symptom onset was also evaluated.
Among the 155 patients enrolled, 42 patients were excluded (no FLAIR image, n=23; negative DWI, n=11; insufficient image quality or clinical information, n=8). Thus, 113 patients were included for final analysis: mean age was 67.0±12.6 years and 61 (54.0%) were men. The initial National Institutes of Health Stroke Scale score was 4±5. Patients received MRI at 7.0±7.1 hours from symptom onset. Fifteen patients (13.3%) received tissue plasminogen activator before and during MRI.
In pair A, the inter-rater agreement of DWI-FLAIR mismatch was 0.538 with conventional images and 0.754 with color-coded images (Figure 2). The percent agreement was significantly different between conventional and color-coded images (69.9% versus 85.8%, respectively; P=0.004). The proportion of patients categorized as subtle change was higher with conventional than with color-coded images (38.1% versus 25.7%, respectively; P=0.01). The inter-rater agreement was increased by color-coded image, especially in patients with obvious change (Figure 2).
Of the 113 patients, 34 patients (30.1%) demonstrated discrepant ratings with conventional images. It was resolved in 28 of 34 patients (82.4%) with color-coded images. Those 28 patients demonstrated a high proportion of cardioembolic stroke (46.4% versus 21.2%; P=0.02) and cortical lesion location (42.9% versus 17.6%; P=0.04) compared with other patients (Table I in the online-only Data Supplement). Color-coded FLAIR image decreased the proportion of obvious and subtle FLAIR signal change, especially in patients beyond the time window of thrombolysis (Figure 3). Therefore, the positive predictive value of DWI-FLAIR mismatch predicting lesion <4.5 hours of stroke onset was 85.3% and 71.9% with conventional FLAIR images and 95.7% and 82.1% with color-coded FLAIR images, respectively, by each rater.
Color-coded FLAIR image, a simple and an easily applicable method by clinicians, increased the inter-rater agreement of FLAIR signal change, especially in patients with cortical lesions. These focal cortical lesions were previously disregarded from conventional FLAIR images,5 but may increase hemorrhagic risk after thrombolysis. Consistent with our results of conventional FLAIR images, recent studies that included patients with similar stroke severity to the present study demonstrated a moderate inter-rater agreement (k=0.57 and 0.48).2,8 Semiautomatic measurement of signal intensity ratios inside a circular region of interest did not significantly improve inter-rater agreement.8 This may be related with the fact that the pathological status of infarcted tissue is heterogeneous, which was well demonstrated by color-coded FLAIR image.9
Additionally, color-coded FLAIR images decreased the proportion of patients graded as subtle change. Previously, the inter-rater agreement and the correlation between time from stroke onset and FLAIR signal change was low in patients graded as subtle FLAIR change.7 Therefore, reducing patients rated as subtle change may help increase the reliability of DWI-FLAIR mismatch in the clinical field. Finally, excluding patients >4.5 hours of symptom onset is important to avoid unexpected side effects of reperfusion therapy in the ongoing clinical trials with patients with unknown-onset stroke. Therefore, the high specificity and positive predictive value of color-coded FLAIR images may have advantage in safety concerns of thrombolysis (Table II in the online-only Data Supplement).
Limitations stemming from the possibility of overestimation and estimation bias occurring from learning effect cannot be excluded. However, restricting the area for evaluation to the DWI lesion and the previously described various efforts to minimize the learning effect may help resolve these issues.
Color coding of FLAIR images, an easily applicable technique, increased the inter-rater agreement of FLAIR signal change. It may be potentially beneficial in informing the decision regarding thrombolysis in patients with unknown-onset stroke.
Sources of Funding
This study was supported by grants from the Korea Health Technology R&D Project, Ministry for Health and Welfare, Korea (HI12C1847), and the Asan Institute for Life Sciences (No. 2013–365), Seoul, Korea.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.114.006515/-/DC1.
- Received June 19, 2014.
- Accepted July 11, 2014.
- © 2014 American Heart Association, Inc.
- Thomalla G,
- Cheng B,
- Ebinger M,
- Hao Q,
- Tourdias T,
- Wu O,
- et al
- Kim BJ,
- Kim HJ,
- Lee DH,
- Kwon SU,
- Kim SJ,
- Kim JS,
- et al
- Kang DW,
- Sohn SI,
- Hong KS,
- Yu KH,
- Hwang YH,
- Han MK,
- et al
- Song SS,
- Latour LL,
- Ritter CH,
- Wu O,
- Tighiouart M,
- Hernandez DA,
- et al
- Cheng B,
- Brinkmann M,
- Forkert ND,
- Treszl A,
- Ebinger M,
- Köhrmann M,
- et al
- Galinovic I,
- Puig J,
- Neeb L,
- Guibernau J,
- Kemmling A,
- Siemonsen S,
- et al