Systematic Review, Critical Appraisal, and Analysis of the Quality of Economic Evaluations in Stroke Imaging
Background and Purpose—This study reviews the quality of economic evaluations of imaging after acute stroke and identifies areas for improvement.
Methods—We performed full-text searches of electronic databases that included Medline, Econlit, the National Health Service Economic Evaluation Database, and the Tufts Cost Effectiveness Analysis Registry through July 2012. Search strategy terms included the following: stroke*; cost*; or cost–benefit analysis*; and imag*. Inclusion criteria were empirical studies published in any language that reported the results of economic evaluations of imaging interventions for patients with stroke symptoms. Study quality was assessed by a commonly used checklist (with a score range of 0% to 100%).
Results—Of 568 unique potential articles identified, 5 were included in the review. Four of 5 articles were explicit in their analysis perspectives, which included healthcare system payers, hospitals, and stroke services. Two studies reported results during a 5-year time horizon, and 3 studies reported lifetime results. All included the modified Rankin Scale score as an outcome measure. The median quality score was 84.4% (range=71.9%–93.5%). Most studies did not consider the possibility that patients could not tolerate contrast media or could incur contrast-induced nephropathy. Three studies compared perfusion computed tomography with unenhanced computed tomography but assumed that outcomes guided by the results of perfusion computed tomography were equivalent to outcomes guided by the results of magnetic resonance imaging or noncontrast computed tomography.
Conclusions—Economic evaluations of imaging modalities after acute ischemic stroke were generally of high methodological quality. However, important radiology-specific clinical components were missing from all of these analyses.
See related article, p 660.
Patients with stroke often require extensive inpatient and outpatient care. In Canada, the annual cost of stroke care was recently estimated at Cdn $2.7 billion.1–3 These high human and economic costs may be mitigated by decreasing stroke-related morbidity and mortality. Optimal acute ischemic stroke (AIS) management includes timely access to neuroimaging (computed tomographic [CT] or MRI scanning) as well as thrombolytic therapy when indicated.4,5 Baseline neuroimaging is used to rule out intracerebral hemorrhage and confirm the presence of infarction.6
In general, the use of economic evaluations to assess medical imaging technology is in its infancy. However, radiological interventions for cerebrovascular disease are 1 of the top 2 clinical areas in which imaging-focused economic evaluations are being undertaken.7 Although the number of economic evaluations within medical imaging has increased, the quality of these studies has been called into question.7–9 Despite the publication of guidelines for health economic studies, there has not been a concomitant increase in the quality of published, imaging-specific economic evaluations.7,10,11
High-quality economic evaluations can be powerful tools that generate robust estimates of the impact of a heath technology on health outcomes and costs. General methodological issues, and factors specific to patients with stroke, need to be considered when assessing the validity of economic evaluations of imaging after stroke. These include the model structure of the economic evaluation, the data chosen to populate the model, as well as the accuracy of stroke outcome measures and outcomes specific to the use of certain imaging modalities.12 Flaws in any of these areas will hamper the results of the analyses and potentially misinform decisions concerning economic impact and health outcomes. A review of the quality of economic evaluations can help elucidate common weaknesses within economic evaluations and identify areas for improvement. Three systematic reviews of economic studies in stroke management, which included quality assessments of eligible studies, have been performed.13–15 However, no review has assessed economic evaluations of imaging modalities for patients with acute stroke, to inform thrombolytic therapy decisions.15 Therefore, we systematically reviewed and assessed the quality of the economic evaluations of medical imaging in patients with stroke-like symptoms.
Materials and Methods
A systematic review of studies published from 1950 to July 2012 was performed. The following electronic databases were searched for relevant economic evaluations: MEDLINE (1950 to July Week 4, 2012), EMBASE (1980 to July Week 4, 2012), OVID HealthSTAR (1966 to July 2012), CINAHL (1981 to July 2012), the Tufts Medical Center Cost Effectiveness Analysis Registry (1976 to July 2012), the British National Health Service Economic Evaluation Database (July 2012), and Econlit (1969 to July 2012). The search strategy was developed in conjunction with a library scientist and included the following keywords: stroke*; cost*; or cost–benefit analysis*; and imag* or scan* (Appendix A in the online-only Data Supplement). The EMBASE search was undertaken using guidelines on how to optimize searches for economic studies in EMBASE.16 Relevant journals were manually searched in addition to the citation lists of reviews of economic evaluations and other relevant articles. A search of the gray literature was also performed, using a gray literature checklist.17
Studies published in any year and any language were included if they performed a comparative analysis of both costs and effects of ≈2 different imaging strategies that are commonly used to manage patients with stroke-like symptoms or identify patients for lytic therapy. Studies of adults and children were eligible. Studies were excluded if they did not report primary data or if they reported solely costs or outcomes. Duplicate citations (n=495) were deleted using Papers 2 (version 2.2.10, Mekentosj, Aalsmeer, The Netherlands). Two reviewers independently applied these criteria to determine study eligibility. Discrepancies were resolved by consensus between the 2 reviewers.
Currently, there is no consensus about the best instrument for assessing the quality of an economic evaluation. We chose the first-published instrument (1996), designed specifically for economic modeling studies and economic submissions to the British Medical Journal—one of the most commonly used for assessing economic evaluation quality.18 An overall quality score is assigned to each study ranging from 0 (low, 0%) to 35 (high, 100%). Two reviewers independently evaluated the studies. The reviewers were blinded to the study identifiers. Discrepancies were resolved through consensus.
Data were abstracted from each study and included study characteristics (eg, author[s], publication year, patient population), items in the quality assessment tool, and key aspects of the analysis not included in the quality assessment tool such as the actual incremental cost-effectiveness ratio (ICER).
A total of 568 nonduplicate articles were identified from the literature search and screened for inclusion. After reviewing the titles and abstracts of each article, 25 articles were selected for full-text review. Twenty of these articles did not meet the study inclusion criteria, and 5 articles were included in this systematic review (Figure).
The articles studied populations from the United States (n=3) and the United Kingdom (n=2). All studies used noncontrast CT (NCCT) as the base case imaging modality to select patients for intravenous thrombolysis.19–23 One study used CT perfusion (CTP),23 and another used MRI as the comparator19 (Table 1). The third used both CTP and MRI as comparators, but not in combination.20 The fourth study used multimodal CT (defined as the combination of NCCT, contrast-enhanced CT, CT angiography [CTA], and CTP) as the comparator.21 The fifth study focused on NCCT only, with the base case being NCCT administered at 48 hours and the comparators being NCCT used at different time points after symptom onset, and in patients with different comorbidities.22
Of the 5 studies reviewed, 4 did not explicitly publish ICERs but provided adequate data for ICERs to be calculated20–23 (Table 2). In 3 studies, NCCT was dominated by ≥1 proposed alternative imaging strategy; that is, the alternative was both cheaper and more effective.20,21,23 The study that compared NCCT at 48 hours to 12 other NCCT strategies determined that the scan-all-patients-immediately strategy delivered a lower cost but lower benefit.22 Another study that compared NCCT with MRI with NCCT alone showed that both the cost and benefit of the combined imaging strategy were higher.19 Finally, when NCCT with MRI was compared with NCCT alone in another study, the overall cost of using NCCT with MRI was cheaper yet demonstrated an identical effect to NCCT.20 The ICERs of the dominant imaging strategies ranged from −£500 to −$257,250, and the ICERs of the nondominant imaging strategies were $1020 and $286,000 per quality-adjusted life-year (QALY).
Three studies demonstrated that NCCT was dominated by 3 alternative imaging strategies such as (1) NCCT+CTP23; (2) NCCT+CTP20; and (3) multimodal CT (MMCT)21 (ie, they had greater effectiveness and lower cost; Table 3). Jackson et al23 determined that the strategy of NCCT+CTP was less costly than NCCT alone because of a reduction in use of thrombolysis. The addition of CTP demonstrated greater benefit because CTP was better able to exclude patients who were at higher risk of secondary intracranial hemorrhage (ICH) and better include patients who could benefit from thrombolysis at >4.5 hours postsymptom onset. However, the outcomes used in the economic evaluation were based on the results of studies that used MR perfusion, not CTP imaging.23 Earnshaw et al20 also compared NCCT to (NCCT+CTP) and demonstrated a similar reduction in cost similar to the other study, which compared these modalities but a greater incremental benefit (0.12 QALYs versus 0.0005 QALYs). They attributed the domination of NCCT+CTP over NCCT alone to the better selection of patients for thrombolysis. Finally, Young et al21 reported the greatest difference in cost of MMCT compared with the other studies included in the review ($2058 over a lifetime horizon in patients who received intra-arterial procedures). The unit cost used for stroke treated with intra-arterial thrombolysis was much greater than that for those treated with intravenous thrombolysis ($33,500 versus $17,200). The increased costs and benefits associated with the addition of MMCT were offset by improved selection of patients for intra-arterial procedures, resulting in fewer patients receiving intra-arterial thrombolysis and better outcomes for those who were selected by MMCT to receive it.
The median quality score of the 5 economic evaluations was 27 (range=23–29) with a median percentage score of 84.4% (range=71.9% to 93.5%; Table 4). Four studies used decision trees,19,20,22,23 and 1 study incorporated Markov modeling to perform the economic evaluation.21 Three of the 5 studies provided justification for the choice of model to generate the CEA estimates.20–22 The time horizon in these studies ranged from 5 years (2)22,23 to lifetime (3).19–21 One study did not use a discount rate.22
All studies clearly outlined their primary outcome measures, and all used the modified Rankin scale (mRS) score to report functional outcomes. However, the use and definitions of mRS-based outcomes differed among the studies. Three studies evaluated CTP as a comparator with NCCT.20,21,23 One study assumed that the probabilities of outcomes for patients receiving CTP were equivalent to those of patients receiving penumbra-based MR imaging.23 Another assumed that the probabilities associated with CTP were equivalent to those for patients in stroke trials that selected patients for tissue plasminogen activator therapy using NCCT or MRI (but not CTP).21 Two studies included an adverse event transition state of secondary ICH in patients selected for thrombolytic therapy with CTP but assumed the transition probability of secondary ICH to be similar to those patients selected via MRI.20,23 Sensitivity analyses of 2 studies demonstrated that their cost–benefit estimates were sensitive to changes in transition probabilities for CTP to select patients for tissue plasminogen activator therapy20,21 (Table 5).
None of the studies using MRI as a comparator added probabilities for incomplete MRI secondary to bail out nor accounted for the delay in time to therapy while MRI was performed.23 Additionally, although the incidence of contrast-induced nephropathy (CIN) among patients with stroke receiving CTP or CTA is low, CIN can significantly add to the length of patient stay and concomitant costs of care.24 One study omitted any CIN transition state for patients receiving CTP,21 another study included probabilities for patients not able to tolerate contrast media,19 but included no probabilities for patients actually incurring CIN. Yet another study included a transition state for patients experiencing CIN, and their results were sensitive to changes in this variable.23
Patients diagnosed with AIS are at risk for recurrent stroke. Only 1 of the 5 analyses considered the rates of stroke recurrence.23 Furthermore, although all models in the review incorporated tissue plasminogen activator treatment, 2 of the articles failed to include ICH states after thrombolysis within their models.21,22
Analysis and Interpretation of Results
Sensitivity analyses should be explicitly described, with justification for the variables selected and the ranges over which they were varied. All 5 studies performed sensitivity analyses, but 2 studies failed to provide justification for the variables selected.22,23 One study performed 1-way sensitivity analyses only,22 while the remainder performed 1-way, probabilistic, and scenario sensitivity analyses. Three studies included sensitivity analyses around imaging factors: (1) sensitivity and specificity of CT imaging to detect AIS22; (2) sensitivity and specificity of CTA to detect AIS21; (3) incidence of contrast-induced nephropathy among CTP patients23; and (4) time to perform and interpret an MRI scan.19 One study used the author’s expert opinion to estimate the ranges of sensitivity analyses for some variables,21 although the remaining ranges were estimated from published literature.
There is increasing demand from healthcare policy makers and providers for rigorous economic evidence to inform decisions around the judicious allocation of limited resources. The introduction of advanced diagnostic imaging techniques has resulted in an increased demand for their use and a concomitant proliferation in related expenditures.10 Economic evaluations of stroke management are especially valuable because stroke is a condition that has among the highest rates of morbidity and mortality in the Western world, which are expensive.25 It is important that these economic evaluations are of high quality.7,10
We systematically reviewed and critically appraised the quality of the 5 published economic evaluations of acute stroke imaging. Although they were generally of high methodological quality (scores ranging from 71.9% to 93.5%) according to a widely used economic evaluation tool, many improvements can be made, so that the models used are more accurate representations of the reality of imaging and the care of patients with AIS.
The results of an economic evaluation may be impacted by any number of methodological flaws. One of the major flaws, which can result in serious study bias, is the failure to include all relevant comparators.26 Each of the included studies compared NCCT with ≥1 neuroimaging comparators that are commonly used to select patients for AIS treatment. These included contrast CT, CTP, MMCT, and MRI. However, no study included all possible, reasonable comparators with NCCT. The ideal neuroimaging model for patients with stroke-like symptoms should include NCCT compared with each currently available (and commonly used) neuroimaging modality used alone, as well as plausible combinations of modalities.
Important considerations when assessing the cost-effectiveness of imaging modalities after acute stroke include the time between symptom onset and the imaging modality, and the impact of the image results on subsequent treatment (particularly thrombolysis) and clinical outcomes. Other important considerations are the ability of the imaging modality to depict pathological changes in anatomy and function versus normal anatomy and function; and whether the images are of high enough quality to be useful.27 These factors should be considered in any future economic evaluations of stroke imaging, specifically: (1) the sensitivity and specificity of any imaging modality used; (2) the proportion of imaging studies that are interpretable; (3) the additional time taken to acquire, process, and interpret multimodal imaging techniques.
CTP is a relatively new, comparatively inexpensive, and efficient imaging technique that can distinguish infarcted from salvageable brain tissue in patients with suspected AIS.28 It is not surprising, then, that 3 of the studies used CTP as a comparator.11,20,21,23 However, as expected for a relatively new technology, there are no high-quality data available about the outcomes after CTP imaging in patients with AIS. In the absence of such data, each economic evaluation assumed that the CTP-related transition probabilities after imaging were equivalent to those of patients managed with either NCCT imaging or MRI. These assumptions are concerning because sensitivity analyses in 2 of these studies demonstrated that the cost-effectiveness estimates of CTP were sensitive to changes in CTP imaging transition probabilities.21,29 It is apparent that these models are hampered by these missing data, and that CTP-specific outcome data are needed to improve these models.
Several imaging-specific factors may influence economic evaluations, including deleterious effects of contrast media, long-term effects of radiation exposure among those patients who received CT imaging, and claustrophobia. First, a problem among evaluations that included CT or MRI comparators was the lack of consideration of renal pathology because of contrast-induced nephropathy and nephrogenic systemic fibrosis, respectively. Although uncommon, these are clinically important complications and should be included in any models that assess CT with iodinated contrast or MRI with gadolinium contrast. Second, only 1 of the 2 studies assessing MRI comparators considered MRI bail-out secondary to claustrophobia.19 Because an estimated 1% to 30% of patients experience MRI claustrophobia, this is a potential driver of cost-effectiveness among patients who receive MRI and should be included in these analyses.30,31
Hemorrhagic transformation and stroke recurrence are 2 clinical outcomes among patients with AIS, which are important enough to be included in economic evaluations of interventions among these patients. From trial data, 4% of patients with AIS who received thrombolysis have a symptomatic ICH,32 and 14% experience recurrent stroke within 90 days of the index stroke.28 Although stroke recurrence was a commonly reported state in a review of the quality of economic analyses of stroke management,15 only 1 study in our systematic review took stroke recurrence into consideration.23 Similarly, 2 studies failed to take into account the adverse costs and effects of ICH secondary to thrombolysis therapy.21,22
Each of the economic evaluations included in this study compared the use of NCCT alone with that of other neuroimaging strategies. In 3 studies, NCCT was dominated by ≥1 proposed alternative imaging strategy.20,21,23 Because the studies included in this review did not use CTP-specific data, new economic evaluations of these imaging strategies should be performed using high-quality, CTP-specific data. Results from these future studies could help to inform policy and determine whether all patients with suspected AIS should receive CTP imaging in conjunction with NCCT.
To our knowledge, this is the first review and critical appraisal of the quality of economic analyses in AIS imaging. Our literature search strategy was thorough, and we are confident that all relevant economic evaluations of AIS imaging have been captured. A limitation of our study is the use of a popularly used yet unvalidated tool for quality assessment of economic evaluations. We encountered difficulties with this tool during the assessment of the studies related to ambiguity of the wording of certain tool items. Future studies would benefit from the use of a quality assessment tool that is developed using a consensus-based process to select the most relevant domains and items that best indicate quality in economic analyses, such as the newly published CHEERS tool.33
In conclusion, economic evaluations of imaging modalities in patients with AIS are of relatively high quality with respect to a published economic modeling guideline.34 However, because of several clinical and radiological limitations, these studies need improvement, and their results may be misleading. Future economic evaluations of stroke imaging should compare all reasonable neuroimaging modalities, incorporate imaging modality sensitivities/specificities, include CTP-specific outcome data, and should integrate the probabilities of hemorrhagic transformation and recurrent stroke after treatment. The results of this review should help to guide the development of high-quality economic models in stroke imaging that will provide valuable information for those interested in the most cost-effective use of AIS imaging.
Sources of Funding
This work was supported by The Canadian Stroke Network; the Heart and Stroke Foundation of Canada; and the Canadian Institutes of Health Research.
This study originated from the University of Toronto Department of Medical Imaging and Institute of Health Policy, Management, and Evaluation.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.113.004027/-/DC1.
- Received November 5, 2013.
- Accepted December 6, 2013.
- © 2014 American Heart Association, Inc.
- 4.↵Collaboration S. Organised inpatient (stroke unit) care for stroke. Cochrane Database Syst Rev. 2007;CD000197.
- Latchaw RE,
- Alberts MJ,
- Lev MH,
- Connors JJ,
- Harbaugh RE,
- Higashida RT,
- et al
- Holloway RG,
- Benesch CG,
- Rahilly CR,
- Courtright CE
- Evers SM,
- Ament AJ,
- Blaauw G
- 17.↵Grey Matters: a practical search tool for evidence-based medicine. Canadian Agency for Drugs and Technologies in Health Web site. http://www.cadth.ca/en/resources/finding-evidence-is/grey-matters. Accessed December 22, 2013.
- Drummond MF,
- Jefferson TO
- Earnshaw SR,
- Jackson D,
- Farkouh R,
- Schwamm L
- Wardlaw JM,
- Seymour J,
- Cairns J,
- Keir S,
- Lewis S,
- Sandercock P
- Jackson D,
- Earnshaw SR,
- Farkouh R,
- Schwamm L
- Hopyan JJ,
- Gladstone DJ,
- Mallia G,
- Schiff J,
- Fox AJ,
- Symons SP,
- et al
- Lloyd-Jones D,
- Adams RJ,
- Brown TM,
- Carnethon M,
- Dai S,
- De Simone G,
- et al
- Dantendorfer K,
- Wimberger D,
- Katschnig H,
- Imhoff H
- Drummond M,
- Weatherly H,
- Ferguson B