Innovations in Stroke
The Use of PROMIS and NeuroQoL Scales in Clinical Stroke Trials
There are significant limitations with the use of modified Rankin Scale (mRS) score as the primary outcome in clinical stroke trials. We propose the exploration of the Patient-Reported Outcome Measure Information System (PROMIS) and Quality of Life in Neurological Disorders (NeuroQoL) tools as outcome measures in clinical trials and provide preliminary data on the distribution of these measures in a cohort of patients with ischemic stroke.
The mRS, a clinician-reported measure of global disability,1 is a commonly used primary outcome in clinical stroke trials. It is a 1-item measure ranging from grades 0 to 5 with 0 representing no symptoms and 5 representing severe disability. A score of 6 is often used to represent death. Although its beneficial attributes include simplicity and various modes of administration and demonstrated construct and convergent validity,2 there are also limitations when using mRS. There can be substantial interobserver variability in scoring, especially between scores 1 and 4,3 which is of key importance for stroke trials that dichotomize outcomes. In addition, the mRS is heavily weighted towards mobility4 and does not specifically delineate several other important domains of health that are frequently affected by stroke, such as cognition, fatigue, or ability to participate in social roles. The traditional approach to analyzing mRS in clinical trials is to dichotomize the score, which reduces the sensitivity to detect change. Shift analysis has the potential to increase power to detect differences compared with dichotomization. However, to allow adjustment for covariates and estimate an effect size, shift analyses must be done using ordinal regression, which requires that the data meet proportionality assumption.5 Importantly, because the mRS has only 6 levels, the ability to detect meaningful change and differences in outcomes among treatment groups is limited, even with the use of shift analysis. The use of continuous scales may provide more power to assess differences between treatment groups.5
The importance of obtaining outcomes information directly from patients is gaining recognition. Patient and provider perceptions may differ and standard outcome measures do not often capture many aspects of outcomes relevant to patients.6 As such, the Food and Drug Administration has a renewed interest in the use of patient-reported outcome measures (PROMs) in clinical trials7 to include the patient’s voice in the assessment of interventions.
It would be a powerful advance for clinical stroke research to incorporate tools into outcomes measures that include multiple domains of health important to patients, can be efficiently and reliably captured through patient report, and could increase statistical power to detect differences between treatment groups. The use of the PROMIS or NeuroQoL system of tools may offer such a solution. PROMIS is a patient-reported system based on item response theory that uses a continuous scale to efficiently measure patient health status for different domains of health8 in individuals with a wide range of diseases and demographic characteristics. The mean of each scale is 50, reflecting the mean of the general population, and the standard deviation (SD) is 10. Scores have a normal distribution allowing for simpler and more powerful parametric statistical models.9 Domains are assessed through patient questions from item banks, reducing ceiling and floor effects. The use of computer adaptive testing allows the system to query patients based on their previous responses until a prespecified level of precision is reached, typically within 3 to 4 questions, which significantly reduces patient burden. NeuroQoL is a sister suite of tools developed specifically for use in neurological populations10 with overlapping item banks with PROMIS. Stroke was one of the 5 neurological conditions used to perform item pool testing and score calibrations. NeuroQoL uses the same standardized scoring system as PROMIS, and the scales have been cocalibrated to the corresponding PROMIS scales, allowing measurement of both scales along a common metric (http://www.prosettastone.org/measures/Neuro-Qol).
Extensive qualitative and quantitative evaluation of both PROMIS and NeuroQoL domains ensures that they capture a valid, reliable measure of the outcome of interest. Short forms are also available for PROMIS and NeuroQoL tools, consisting of static sets of questions, for situations where computer adaptive testing is unavailable or when providers prefer to deliver the same set of questions to all patients. The use of a continuous linear scale and normalized distribution may potentially increase sensitivity to change11 and improve statistical power to detect differences across treatment groups. As an added benefit, PROMIS and NeuroQoL tools can be used across conditions, making cross-population comparisons feasible. Electronic administration of PROMs is gaining popularity. Many PROMIS short forms are available in Epic’s electronic health record (Epic Systems, Verona, WI), and they are considering incorporating PROMIS computer adaptive testing s in a future release (Richard Gershon, PhD, personal communication, August 27, 2015).
We investigated the pattern and distribution of PROMIS scores in ischemic stroke patients seen in the Cleveland Clinic cerebrovascular ambulatory clinic through a cross-sectional analysis of data from several PROMIS domains. Patients in the cerebrovascular clinic routinely complete PROMs on electronic tablets at the time of the visit or through the patient portal to the electronic health record (Epic) before arriving for their visit with results immediately available within the electronic health record.12 The PROMIS Fatigue and PROMIS Physical Function (PROMIS PF) have been routinely collected as part of this system since September 2012. The PROMIS domains anxiety, satisfaction with social roles, sleep disturbance, and pain intensity were added in February 2015. Depression is assessed with the patient-reported Patient Health Questionnaire-9. These scores are then cocalibrated on the PROMIS metric,13 which provides an equivalent PROMIS depression score. The mRS is also completed by the provider at the visit.
The patient cohort consisted of all patients with diagnosis of ischemic stroke who had a completed PROMIS score and mRS between September 14, 2012, and June 16, 2015. If data from >1 encounter was available, data from the first encounter was used.
In this analysis, for all PROMIS scores, higher scales indicate better functioning. The frequency distribution of PROMIS PF was calculated overall and according to mRS level. mRS scores of 4 and 5 were combined because of low numbers within these levels. Spearman correlation coefficients were calculated between mRS and the PROMIS scales, and mean scores of the PROMIS scales stratified by mRS level were computed.
There were 3762 visits by patients with ischemic stroke with a completed mRS and PROMIS PF scale, comprising 2431 unique patients (Table). Mean PROMIS PF score was 40.6 (SD=11.3), and median mRS score was 1 (interquartile range [IQR] 1–2). At the first visit during the study period, 541 (22.3%) patients had a normal mRS score (=0), demonstrating a significant ceiling effect. In contrast, 12 (0.5%) patients scored at ceiling of the PROMIS PF. It had a normal distribution overall and within most strata of mRS (Figure 1), suggesting the potential for PROMIS PF to more finely discriminate among patient groups.
Between February 17 and June 16, 2015, 323 patients completed the additional PROMIS scales (Table). Scores of the 7 PROMIS scales were lower with successively higher mRS levels, although difference in scores across mRS levels varied (Figure 2). The PROMIS PF had the highest overall correlation with mRS (ρ=−0.60), the PROMIS Sleep Disturbance had the lowest (ρ=−0.19). At higher levels of disability in functional activities as defined by mRS levels, patients reported worst physical function and social satisfaction, whereas sleep and pain were relatively less impacted. These findings show the potential of PROMIS tools to augment information on a patient’s health status that is not captured by the mRS. Importantly, the domains of patient’s perceived health do not decline to the same degree at increasing severities of mRS, further supporting the value of assessing individual health domains to allow detection of differential changes within different aspects of health.
Limitations and Next Steps
Several questions need to be addressed before PROMIS or NeuroQoL tools can be successfully used in clinical stroke trials. Evidence of acceptable psychometric properties is a prerequisite for inclusion of outcomes measures into clinical research, which includes content validity, test–retest reliability, and ability to detect change (responsiveness) and to discriminate between groups. Many of these items have already been addressed in stroke with the NeuroQoL tools.10 Determination of outcome domains that are relevant to patients, a form of content validity, is a requisite for the Food and Drug Administration to use PROMs for approval and labeling claims. Identification of the minimally important difference, which reflects a score difference large enough to have implications for a patient’s care, is also critical. The minimally important difference of outcome measures typically estimated to be one-half the SD14 has been found to be accurate in an analysis of PROMIS scales designed for use in patients with cancer.15
The use of PROMs will add a level of complexity when evaluating outcomes of care because they may be impacted by the patient’s environment, social support, mental outlook, and their self-defined standards for health.16 Further exploration should be performed on the impact of various factors on patient report of health outcomes to determine whether and how to incorporate this information into the interpretation of clinical stroke trial results.
Stroke patients may have cognitive or other functional limitations preventing them from completing PROMs. Proxy assessment may substitute for patient self-assessment in these cases. Preliminary data with NeuroQoL scales in stroke patients suggest comparable performance between patient and proxy responses in group-level analyses for moderate-high functioning patients with stroke.17 Patients may also die or become too ill to complete patient-reported questions. A method to account for dropouts because of death or worsening health has been described.18
An imbalance in prognostic variables between treatment groups, including premorbid health status, may result in incorrect results.5 In acute stroke trials that use mRS as the outcome, imbalance in premorbid functioning is reduced somewhat with the common enrollment requirement of a premorbid mRS score <2 or 3. Imbalance in premorbid status may have more consequence when using outcomes measures that allow finer discrimination in health status than the mRS. More research will be useful to determine the importance of accounting for differences in premorbid states of health in the various domains used in the assessment of outcome with the PROMIS or NeuroQoL tools, and if salient, how best to accomplish this.
To explore the use of PROMIS or NeuroQoL domains in clinical research in stroke, prospective observational studies of a broad population of patients should be performed that entails completion of the selected PROMIS/NeuroQoL tools at specific time points after stroke. This will provide important information on the range and variability in scores and patterns of change over time across patients of different ages and severities of impairment, which will inform the use of these tools in clinical trials. Inclusion of PROMIS/NeuroQoL tools as outcomes in clinical stroke trials will provide direct information on their ability to detect differences in patient outcomes across treatment groups. Ideally, power and sample size calculations would account for the statistical analysis of one or more of these PROMs as key secondary outcomes.
The potential benefits of including PROMIS/NeuroQoL tools as outcomes in stroke clinical trials are compelling and include the ability to measure outcomes on a continuous scale potentially improving their power to detect change, the incorporation of patient viewpoints, and the ability to assess outcomes across multiple domains affected by stroke. Recent advances in the technical capabilities to electronically administer PROMs make this feasible. Further research is warranted to evaluate several unresolved questions related to the use of PROMIS/NeuroQoL tools in stroke clinical trials.
Srividya Ramachandran, PhD, provided editorial assistance for this manuscript.
- Received September 1, 2015.
- Revision received November 20, 2015.
- Accepted December 8, 2015.
- © 2016 American Heart Association, Inc.
- van Swieten JC,
- Koudstaal PJ,
- Visser MC,
- Schouten HJ,
- van Gijn J.
- Banks JL,
- Marotta CA.
- Quinn TJ,
- Dawson J,
- Walters MR,
- Lees KR.
- Bath PM,
- Lees KR,
- Schellinger PD,
- Altman H,
- Bland M,
- Hogg C,
- et al
- Ali M,
- Fulton R,
- Quinn T,
- Brady M
- 7.↵US Food & Drug Administration. PDUFA V clinical outcomes assessment development and implementation: Opportunities and challenges public workshop. http://www.Fda.Gov/drugs/newsevents/ucm431040.Htm. April 1, 2015.
- 8.↵PROMIS Network. PROMIS - dynamic tools to measure health outcomes from the patient perspective. http://www.nihpromis.org. Accessed June 29, 2013.
- Saver JL.
- Kozlowski AJ,
- Singh R,
- Victorson D,
- Miskovic A,
- Lai JS,
- Harvey RL,
- et al
- Cella D,
- Wang M,
- Wagner L,
- Miller K.