Factors Associated With Time to Site Activation, Randomization, and Enrollment Performance in a Stroke Prevention Trial
Background and Purpose—Multicenter clinical trials attempt to select sites that can move rapidly to randomization and enroll sufficient numbers of patients. However, there are few assessments of the success of site selection.
Methods—In the CREST-2 (Carotid Revascularization and Medical Management for Asymptomatic Carotid Stenosis Trials), we assess factors associated with the time between site selection and authorization to randomize, the time between authorization to randomize and the first randomization, and the average number of randomizations per site per month. Potential factors included characteristics of the site, specialty of the principal investigator, and site type.
Results—For 147 sites, the median time between site selection to authorization to randomize was 9.9 months (interquartile range, 7.7, 12.4), and factors associated with early site activation were not identified. The median time between authorization to randomize and a randomization was 4.6 months (interquartile range, 2.6, 10.5). Sites with authorization to randomize in only the carotid endarterectomy study were slower to randomize, and other factors examined were not significantly associated with time-to-randomization. The recruitment rate was 0.26 (95% confidence interval, 0.23–0.28) patients per site per month. By univariate analysis, factors associated with faster recruitment were authorization to randomize in both trials, principal investigator specialties of interventional radiology and cardiology, pre-trial reported performance >50 carotid angioplasty and stenting procedures per year, status in the top half of recruitment in the CREST trial, and classification as a private health facility. Participation in StrokeNet was associated with slower recruitment as compared with the non-StrokeNet sites.
Conclusions—Overall, selection of sites with high enrollment rates will likely require customization to align the sites selected to the factor under study in the trial.
- carotid arteries
- carotid stenosis
- clinical trial
- endarterectomy, carotid
- multicenter study
- randomized controlled trial
- vascular diseases
See related article, p 2339
Critical for success of large multicenter phase III clinical trials is the identification of clinical sites that can (1) quickly move through the steps required for approval of randomization, (2) rapidly initiate randomization, and (3) enroll a large number of patients to the trial, including women and minorities. To meet this challenge, the leadership of the CREST-2 (Carotid Revascularization and Medical Management for Asymptomatic Carotid Stenosis Trials) created a Site Selection Committee. Similar to the approach taken in many other trials,1–3 the Site Selection Committee established criteria presumed to be associated with capabilities of quick site initiation and large enrollment, and the committee also developed a site questionnaire for the collection of data to evaluate potential study sites. In addition, CREST-2 established linkage to the National Institute of Neurological Disorders and Stroke–funded StrokeNet network, consisting of 25 regional co-ordinating centers designed to maximize efficiencies of conducting high quality, multisite clinical trials in stroke.4,5
Few studies have formally assessed if information available at the time of site selection reliably identifies sites with good performance. Those doing so primarily focused on a single factor as a potential predictor of performance.6–8 At the time of site selection, we could not find a report of a comprehensive assessment of information available with either the time to initiate randomization or the number of patients recruited, which is the goal of these analyses.
The CREST-2 protocol includes a pair of randomized clinical trials.9 One trial compares outcomes of ≈1240 patients randomized to carotid endarterectomy (CEA) plus intensive medical management versus intensive medical management alone (the CEA study). The other trial compares the outcomes of ≈1240 patients randomized to carotid stenting plus intensive medical management versus intensive medical management alone (the carotid angioplasty and stenting [CAS] study).
The Site Selection Committee has evaluated applications from 192 sites during 60 committee meetings from March 5, 2014 to January 9, 2017. The committee identified factors presumed to predict quicker initiation of randomization and higher enrollment: specialty of the site principal investigator (interventional radiology/neuroradiology, neurology, neurosurgery, vascular surgery, or cardiology), site type (private hospital, private office, Veteran’s Affairs medical center, or academic medical center), presence of affiliate or satellite recruiting sites (yes, no), membership in StrokeNet (yes, no), use of a central Institutional Review Board (IRB) (yes, no), presence of the full complement of investigators (full, partial), the reported annual number of CEA procedures performed at the site (<10, 10–25, 26–50, or >50), the reported annual number of CAS procedures performed at the site (<10, 10–25, 26–50, or >50), seeking approval for CAS-only, CEA-only, or both studies, and participation and performance in the CREST (Carotid Revascularization, Endarterectomy versus Stent Trial).10 Potential to enroll women and minorities were additional criteria.
The time between a site being approved as a CREST-2 site and the first randomization can be divided into 2 segments: (1) the period from approval to the time of meeting the requirements to be given authorization to randomize, and (2) the period between receiving authorization to randomize and the time of performing the first randomization. With the approval by the Site Selection Committee, a site may begin the process of completing the prerequisites required for authorization to randomize patients. This includes securing IRB approval, completion of contracts with the Clinical Coordinating Center at Mayo Clinic in Florida, submission and review of information to qualify both surgeons to perform the CEA and the interventionists to perform the CAS, and the training and certification of the clinical center staff on matters of the protocol and data management. With the completion of these steps, the site is given authorization to begin randomization (given the green light letter). Because a site may qualify surgeons but not interventionists, or interventionists but not surgeons, they can be given authorization to begin the CEA or CAS study at different times.
For this report, the time between approval and authorization to randomize, the time between authorization to randomize and the first randomization, and the average monthly enrollment per site were analyzed as indices of site performance. The association between potential factors associated with performance on the 2 time intervals was assessed using standard time-to-event (ie, survival) methodology. The Kaplan–Meier method was used to estimate the proportion with authorization to randomize as a function of time since approval. The significance of the predictive factors was estimated using proportional hazards analysis, with a plan to perform multivariable analysis should >1 factor be significant on univariate testing. An identical approach was used to assess the factors associated with the time between authorization to randomize and first randomization. The association between potential factors associated with recruitment volume was assessed using Poisson regression, which was used to both estimate the recruitment per clinic per month (with 95% confidence intervals [CIs]) and the recruitment ratio between different strata of the potential predictor. Multivariable analysis was used if >1 factor was significant on univariate testing.
As of January 9, 2017, 147 of the 192 sites reviewed were approved for participation in CREST-2. Of these, 122 (83%) completed regulatory and training requirements and were permitted to randomize (ie, received a green light letter); 89 of these 122 (73%) sites had randomized ≥1 patients. The characteristics of all sites, the sites that have been approved for randomization, and those which have randomized ≥1 patients are summarized in Table 1. Over 1713 clinic-months, the 122 permitted sites had performed 437 randomizations, for an overall recruitment rate of 0.26 (95% confidence interval [CI], 0.23–0.28) patients per site per month.
The proportion of clinical sites with authorization to randomize as a function of time since approval is provided in Figure 1A. The minimum time between approval and authorization to randomize was 3.3 months, with 10% of sites receiving authorization in 5.2 months, 25% in 7.7 months, 50% in 9.9 months, 75% in 12.4 months, and 90% in 17.3 months.
None of the factors associated with the time interval between site approval and the authorization to randomize met the P<0.05 criteria. Only prior participation in CREST approached (P=0.055) significance. Compared with sites not in the CREST study, those sites that were in the top half of CREST recruitment had significantly longer time between approval and authorization to randomize (hazard ratio [HR]=0.59; 95% CI, 0.38–0.91), with those in the bottom half of CREST recruitment having nonsignificantly longer time to approval to randomize (HR=0.88; 95% CI, 0.56–1.38). The Kaplan–Meier estimates of the proportion with authorization to randomize shown by CREST participation status is shown in Figure 1B. Because only 1 factor was marginally significant, no multivariable analysis was performed, and because the other results were nonsignificant, they are detailed in the online-only Data Supplement.
Of the 122 sites with authorization to randomize, the proportion performing a randomization as a function of time since that authorization is shown in Figure 2A. The shortest time between authorization to randomize and a randomization was 0.2 months, with 10% of the sites performing a randomization within 0.9 months, 25% within 2.6 months, 50% within 4.6 months, and 75% within 10.5 months.
The only factor significantly associated with the time between approval and randomization was whether the site was approved to randomize to both CEA and CAS, or to CEA-only, or to CAS-only (P=0.014). As shown in Figure 2B, compared with those with authorization to randomize in both the CEA and CAS study, those with authorization for CEA-only were much slower to randomize (HR=0.47; 95% CI, 0.28–0.80) while those with authorization for CAS-only were nonsignificantly slower (HR=0.92; 95% CI, 0.42–2.02). The estimates for the impact of other factors on the time between approval and randomization are provided in Table I in the online-only Data Supplement (all P≥0.15). Because only 1 factor was significant, no multivariable analysis was performed.
Table 2 provides the description between the associated factors and the average monthly recruitment. On univariate analysis, several factors had a large impact on the recruitment rate. Recruitment was higher among those with authorization to randomize in both trials (0.30/mo; 95% CI, 0.27–0.33) or to CAS-only (0.30/mo; 95% CI, 0.22–0.42) compared with those with authorization to randomize only to CEA (0.13; 95% CI, 0.10–0.16). Among the principal investigator specialties, the highest monthly recruitment rates were for interventional radiologists (0.34/mo; 95% CI, 0.24–0.47) and cardiologists (0.33/mo; 95% CI, 0.29–0.33); with sites led by neurologists having almost half the recruitment rate (0.18/mo; 95% CI, 0.14–0.23). Participation in StrokeNet was also associated with slower recruitment (0.17/mo; 95% CI, 0.14–0.21) as compared with the non-StrokeNet sites (0.30/mo; 95% CI, 0.27–0.33). Sites that reported performing >50 CAS procedures per year recruited substantially faster (0.37/mo; 95% CI, 0.31–0.44) than sites reporting <50 CAS procedures per year (all ≤0.23/mo). Those in the top half of recruitment in the CREST trial had a recruitment rate much higher (0.30/mo) than either those in the bottom half of CREST recruitment (0.24/mo; 95% CI, 0.27–0.33) or those not in the CREST study (0.22/mo; 95% CI, 0.18–0.26). Recruitment was much faster in sites that were either a private hospital (0.31/mo; 95% CI, 0.26–0.36) or private office (0.31/mo; 95% CI, 0.26–0.38) compared with academic sites (0.20/mo; 95% CI, 0.17–0.24). There were not significant univariate differences in the recruitment rate (P>0.05) depending on whether sites had affiliates, used a central IRB, had their full complement of investigators, had a larger number of CEA procedures performed at their site annually, or what trial they were approved to conduct by the Site Selection Committee.
The results of the multivariable analysis of the monthly recruitment rate is also provided in Table 2, with the powerful factors being the site type (with VA medical centers showing a dramatic increase), the number of CAS procedures per year at baseline, what trials into which the site has authorization to randomize (CEA and CAS; CEA-only; CAS-only), and CREST participation and performance. In this multivariable analysis, the association between principal investigator specialty and StrokeNet membership became nonsignificant. These changes in the magnitude of association arise from a substantial collinearity between the predictor variables. Specifically, comparing StrokeNet to non-StrokeNet sites, 67% versus 37% were academic, 47% versus 18% had a neurologist principal investigator, 7% versus 28% had >50 CAS procedures annually, 42% versus 17% could randomize only to CEA, and 17% versus 29% were in the top half of recruitment in CREST. As such, the StrokeNet sites had a higher frequency of all the powerful factors associated with slow recruitment, and after adjustment for these differences, there was no difference between the StrokeNet and non-StrokeNet sites. Conversely, univariately the VA medical centers recruited nonsignificantly faster than academic centers (recruitment ratio=1.33; 95% CI, 0.88–2.01). However, they achieved this marginally higher recruitment rate despite having a higher frequency of a powerful factor predicting low recruitment, specifically with 0.0% with >50 CAS procedures compared with 20% in academic centers, and 75% being approved for only CEA randomization compared with 30% in academic centers.
In what we think is one of the first comprehensive assessments, we report time from site selection to randomization, assess factors associated with rapid or slow activation, and examine site characteristics associated with early enrollment. Site activation was prolonged with 1-year required for activation of 3 quarters of the sites, and only prior participation in CREST was near significance (P=0.055) as a predictor of early activation. Initiation of enrollment was also prolonged with 10.5 months required for 75% of the sites to achieve enrollment of ≥1 patients. In contrast to site activation, factors associated with early enrollment were identified and include site and investigator characteristics as will be discussed below.
Complexity of the start-up process impedes identifying factors associated with the selection of sites that initiate randomization quickly. For example, the use of a central IRB may speed IRB approval but would not affect the time to establish contracts or credential clinicians to perform study-approved procedures. Participation in an established network may speed time for contracts but would have no effect on the time to credential clinicians. Importantly, qualifying for randomization is a multistep process, and tactics to accomplish individual steps does not speed the process unless that step is on the critical path to overall approval. For example, credentialing of clinicians to perform the procedures was a remarkable barrier to beginning randomization in CREST-2, so the benefit of quickly moving through IRB and contract approval (factors not on the critical path) was of little value to getting the site quickly approved for randomization
Unlike the absence of factors associated with time-to-randomization, there were a substantial number of factors associated with the average monthly recruitment per site. There seems to be a consistent pattern of associated factors that may inform the selection of sites for future trials. For example, a powerful predictor of high recruitment was an ability to qualify for randomization in the CAS trial. CAS sites may have more incentive to enroll patients because of the limited reimbursement for CAS outside of clinical trials. The low recruitment in sites led by neurologists versus cardiologists is likely a reflection of this same factor because cardiologists would be more likely to qualify for the CAS trial and hence would have an advantage in qualifying for both trials. Those sites reporting a large number of CAS procedures are more likely to qualify for the CAS study. The low recruitment in StrokeNet sites could be similarly associated with low recruitment in sites led by neurologists because 47% of the StrokeNet CREST-2 principal investigators were neurologists (compared with 18% in non-StrokeNet sites). All of this raises the observation that if a trial is studying a particular procedure/technology, expertise and clinical practice in that procedure/technology arena is a paramount qualification for participation.
Apparently, only a few other investigations have examined whether information available as part of the site selection process was predictive of performance in the trial. In a treatment trial of myasthenia gravis, the mean time for sites to achieve regulatory approval was reported to be shorter for sites in the United States (9.7±0.7 months) than for non-US sites (13.4±1.0 months).11 The ALLHAT (Antihypertensive and Lipid Lowering Treatment to Prevent Heart Attack Trial) reported a median number of randomizations of 40 subjects at university sites compared with 79 subjects at Veteran’s Affairs Sites and 37 at private group practices; however, these estimates were not adjusted for the average number of months of recruitment at these different types of sites.8
This study has strengths and weaknesses. Perhaps, the greatest strength is the heterogeneity of sites, allowing an analysis of factors associated with successful site start-up and performance. In addition, the large number of sites in CREST-2 allows reasonable power to detect differences in all 3 indices of performance. For example, with 122 sites receiving authorization to randomize, an HR of 1.66 can be detected with 80% power for a predictor factor that is ≈50% prevalent, and a HR of 1.80 can be detected for a predictor factor that is 25% prevalent. However, there are also shortcomings. This article is offered when 18% (437 of 2480) of the patients anticipated for recruitment have been randomized. We suggest that this is a minor weakness because adequate power to detect differences in the recruitment rates was demonstrated by the large number of significant associations. Site selection bias, particularly for the rate of enrollment analysis, is a potential limitation. The array of potentially unique complexities of each trial and of each trial site are such that some caution should be taken during attempts to generalize the findings to other trials. Researchers planning future studies, of this type, may also consider investigating the association of site, site teams, and site investigators’ cumulative past trial experience with time to activation, randomization, and enrollment performance metrics.
Few studies have reported whether it is possible to preferentially select participating sites that are most likely to succeed in a large multicenter clinical trial. These findings suggest that much more needs to be learned about factors associated with sites that will move rapidly to randomization. However, powerful factors associated with sites’ early recruitment rate in CREST-2 were identified. Collectively, these observations suggest that the selection of sites for high recruitment may need to be targeted and tailored to the treatment under assessment. Targeting sites in this manner could improve the efficiency of future clinical trials.
Sources of Funding
This study was supported by grants U01 NS080168—CREST-2 Clinical Coordinating Center—National Institutes of Health—National Institute of Neurological Disorders and Stroke and U01 NS080165—CREST-2 Statistical and Data Coordinating Center—National Institutes of Health—National Institute of Neurological Disorders and Stroke.
Dr Roubin received royalties from Cook Inc and equity from Essential Medical.
The online-only Data Supplement is available with this article at http://stroke.ahajournals.org/lookup/suppl/doi:10.1161/STROKEAHA.117.016976/-/DC1.
- Received March 1, 2017.
- Revision received May 17, 2017.
- Accepted May 26, 2017.
- © 2017 American Heart Association, Inc.
- Harper BD,
- Zuckerman D
- Broderick JP,
- Palesch YY,
- Janis LS
- Mott M,
- Janis S,
- Koroshetz WJ
- Howard VJ,
- Meschia JF,
- Lal BK,
- Turan T,
- Roubin G,
- Brown RD Jr,
- et al
- Aban IB,
- Wolfe GI,
- Cutter GR,
- Kaminski HJ,
- Jaretzki A 3rd,
- Minisman G,
- et al