Quality of Full and Final Publications Reporting Acute Stroke Trials
A Systematic Review
Background and Purpose—Several studies have shown that the quality of reporting of trials throughout medicine is variable and often poor. We report on the quality of the final reports of randomized controlled trials (RCTs) of drug therapies assessed in acute stroke.
Methods—English-language reports published up to the end of 1996 relating to completed RCTs in acute stroke were identified from electronic searches of the Cochrane Stroke Review Group database of stroke trials and the Cochrane Controlled Trials Register (CD-ROM issue 1, 1997, of the Cochrane Library). Report quality was assessed with the 33 criteria of the CONSORT statement and 53 additional factors relevant to acute stroke or trials in general. Trial quality was also assessed with a 7-point scale.
Results—Up to 1996, 114 RCTs were published which involved 20 536 patients (median, 80; range, 16 to 1267 per trial); 39 (35.5%) of these were published in Stroke. The median total report quality was 40/86 (range, 15 to 61) for all criteria and 19/33 (range, 9 to 29) for the CONSORT criteria alone. Although adequate information was given in the introduction and discussion sections of most reports, insufficient details were given on methods, assignment of patients to treatment groups, statistical analyses, the prevalence of risk factors, and assessment of outcomes. Report quality has improved between 1956 and 1996 (Spearman correlation coefficient [rs], 0.575; 95% confidence interval [CI], 0.439 to 0.685) and was superior in large trials (rs=0.434; 95% CI, 0.274 to 0.571). Although report quality was related to trial quality (rs=0.675; 95% CI, 0.563 to 0.763), it was not related to journal impact factor (rs=0.170; 95% CI, −0.015 to 0.344). Trials with a positive outcome tended to be less well reported than those with a neutral or negative outcome (rs=−0.192; 95% CI, −0.351 to −0.011).
Conclusions—The overall quality of study reports for parallel group RCTs in acute stroke is poor but appears to be improving with time and in parallel with an increase in trial size. Reports often lack detailed information on the methods of randomization, concealment of allocation, and statistical analysis, all factors which can, if undertaken poorly, affect trial results and validity. It is vital that future trials are adequately reported; we believe that authors should follow the CONSORT guidelines and that referees and editors should ensure this happens.
Stroke, in contrast to myocardial infarction, has suffered from an absence of effective and safe treatments despite much testing of vascular and neuroprotective drugs.1 2 Recently, aspirin has been shown to be mildly effective in reducing death and disability in patients with acute ischemic stroke3,4; thrombolysis was shown to be very effective in reducing this combined outcome in one medium-sized trial,5 although in other studies it was found to increase death.6 Hence, in the absence of safe acute treatments with significant efficacy, further trials in stroke are required. However, such trials will have an impact on clinical care only if they are adequately reported so that readers can reasonably evaluate the study in the context of existing information.
Studies in other branches of medicine7 8 9 suggest that the quality of publications describing randomized controlled trials (RCTs) vary considerably in their quality of reporting. As a result, it can be very difficult to compare and contrast apparently similar trials, particularly when designing further trials and performing systematic reviews. While undertaking several systematic reviews of drugs tested in acute stroke,10 11 12 it became apparent to us that the quality of reporting of stroke trials was also very variable. Hence, the primary aim of this report was to systematically analyze the quality of reports describing acute stroke RCTs, as judged by whether they give a minimum set of information describing the “design, conduct, analysis, and generalizability of the trial.”13 A secondary aim was to assess whether inadequately reported stroke trials were more likely to give a positive result, because it has been shown previously that studies which were poorly reported, particularly with respect to details on randomization and allocation concealment, were more likely to be associated with bias in estimating treatment effect.14
We included reports published up to the end of 1996 and relating to an RCT in acute stroke if they were the full and final trial report, published in English in a peer-reviewed journal, and related to a parallel group trial and had recruited patients within 2 weeks of stroke onset. We excluded studies if they were published in books or as abstracts, had a crossover design, involved an active comparator, or primarily involved patients with transient ischemic attacks or subarachnoid hemorrhage.
Identification of Acute Stroke Trials
Potential trials for inclusion were identified from the Cochrane Stroke Review Group database of stroke trials (held in a Reference Manager file; Research Information Systems, Mac version 2.51) and cross-checked with the Cochrane Controlled Trials Register (CCTR; CD-ROM issue 1, 1997, of the Cochrane Library). The CCTR contains references to 112 308 RCTs identified by contributors to the Cochrane Collaboration from hand-searching of medical journals and electronic searching of MEDLINE and EMBASE. The CCTR was searched with the term “acute stroke.” Information on journal impact quality was obtained from the 1995 edition of the Science Citation Index Journal Citation Reports.
We assessed the quality of the first published report when the results of a trial were reported more than once, as occurred for 2 trials. Two of the authors (F.B. and V.O.) independently assessed each published report with respect to its eligibility for the study. Information was then collected from the report and entered independently into separate Excel spreadsheets (Microsoft; Mac version 5.0a); trials were entered in a random order. The resulting spreadsheets were then manually cross-checked to ensure agreement. Disagreements about whether a trial should be included or interpretation of the information in the report were discussed and, if necessary, resolved by the third author (P.B.).
Report quality was judged on the basis of 86 questions (Table 1⇓). These included 33 questions (based on 21 items) identified by the CONSORT group,13 a derivative of the Standards of Reporting Trials Group15 and the Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature.16 We added an additional 53 questions relevant to clinical trials in general or to acute stroke in particular.
The quality of each trial, judged from the information given in the report, was also assessed with a simple 7-point scale; the emphasis on quality was to assess the likelihood that the primary result of the trial was “correct,” ie, that the drug really did have a positive, neutral, or negative effect on outcome. Six points were related to the chance of bias being introduced by problems with randomization, concealment of allocation, or analysis: (1) the trial was open or placebo controlled (score 1 or 2, respectively); (2) the trial was quasi-randomized or truly randomized (score 1 or 2); (3) allocation was poorly or well concealed (score 1 or 2); (4) the trial was unblinded or single, double, or triple blinded (score 0.5, 1, 1.5, or 2); (5) the treatment and control groups were poorly or well balanced for prognostic baseline variables (score 1 or 2); and (6) the results were analyzed by “per protocol” or “intention-to-treat” (score 1 or 2). The number of subjects was also scored (1 to 4), because large trials are more likely to give an answer that approximates to the truth.
Two sets of analyses were performed, one on all 86 variables and the other on the subset of 33 variables listed in the CONSORT criteria as a sensitivity analysis. Nonparametric descriptors (median or mode, semiquartile range [SQR]) and tests (Spearman rank correlation, Spearman partial correlation coefficient) are given,17 because most of the data are of an ordered categorical type. Analysis was performed with a statistical package (Medistat, PFB Ltd).
We identified 114 trials that fulfilled the inclusion criteria. The first trial was published in 1956, while 1994 was the year when most trials (12 trials; 10.5%) were published. A total of 20 536 patients were studied, with individual trials enrolling between 16 and 1267 (median, 80) patients. Seventy-four trials (64.9%) were single center; 1 trial recruited patients from 75 centers. Thirty nine trials (35.5%) were published in the journal Stroke (impact factor of 3.924 in 1995). The impact factors of other journals containing stroke trials ranged from 0.015 to 22.4; 4 trials were published in journals not listed in the 1995 edition of Science Citation Index Journal Citation Reports.
The total report quality varied between 15 (17%) and 61 (71%) of 86, with a median score of 40 (47%) (Figure 1⇓). Table 2⇓ gives the breakdown of scores for sections within each report. Most trial reports contained adequate information within their introduction and discussion sections. However, insufficient details were given in other sections, especially relating to methods, assignment of patients to treatment groups, statistical analysis, prevalence of risk factors, and assessment of outcomes (Table 2⇓). The report quality, assessed with the CONSORT criteria alone, ranged from 9.0 to 29.0 (out of a maximum score of 33), with a median of 19.0 (SQR, 3.0). The median score for trial quality was 10.5 (SQR, 1.3) but varied considerably between a minimum of 6.5 and maximum of 15.0 (the maximum possible score from 7 questions was 16). While mention of consent procedures was reasonably reported (74; 65%), study approval by a research ethics committee (34; 30%) was infrequently noted.
Report quality was significantly and positively correlated with year of publication (Table 3⇓; Figure 2⇓, top), suggesting that it has improved markedly between 1956 and 1996. Report quality was also related to trial size, whether assessed as the number of patients enrolled or the number of centers. Since recent trials have tended to be larger and to involve more patients and centers (Table 3⇓), partial correlation analysis was used to assess the relationship between quality and year, with size held constant (or “partialed out”); report quality continued to be correlated with year of publication (Table 4⇓). Trial quality has also improved between 1956 and 1996 (Figure 2⇓, bottom) and is higher in large multicenter trials.
Although report and trial quality were strongly correlated (Table 3⇑), there was no association between report quality and journal impact factor, and there was only a weak association between trial quality and impact factor. Partial correlation analysis revealed that report quality had no association with impact factor when trial quality was held constant (Table 4⇑).
An inverse relationship was found between report quality and the main finding of the trial (judged by whether the primary outcome was negative, neutral, or positive); ie, positive trials were less likely than negative studies to be reported well, a finding that was independent of year of publication (Tables 3⇑ and 4⇑).
Separate analyses were performed with the CONSORT criteria alone as a form of sensitivity analysis. Correlation coefficients between the two report quality scores were similar (Table 3⇑) except that the CONSORT report quality score was significantly associated with impact factor.
This systematic analysis of the quality of trial reports for acute stroke studies has revealed that many full and final publications give inadequate information on the design, execution, and interpretation of the study. Since we included only trials published in peer-reviewed journals, it is likely that the quality of all reports of acute stroke trials (ie, including those published in non–peer-reviewed journals) will be even lower. The main reason for undertaking trials is to inform and alter medical practice, and hence it is surprising that trialists often pay scant attention to reporting their study. Readers of trial reports have a right to better information, since poor-quality reporting may not only cover up a poor trial design but may also mislead readers as to the importance of the results. Furthermore, it is unethical to expose patients to unproved (and potentially hazardous) treatments if the final study report has no effect on medical knowledge because it is of poor quality and wastes the value of the results.
However, the quality of reporting is improving with time, probably for a number of reasons: (1) trials are becoming larger and therefore involving more authors, (2) authors may be gaining more experience and skill at writing, and (3) referees and journals may be demanding better quality in reports. Recently, a number of journals (including British Medical Journal, Journal of the American Medical Association, and The Lancet18) have adopted the CONSORT recommendations, although it is too early to assess how this has altered the quality of reports. Because the adoption of CONSORT by medical journals only commenced during 1996, it will have had a very marginal effect on our findings.
Our review of report quality has 3 important weaknesses that need to be discussed. First, criteria within both the CONSORT and extended groups were unweighted for scoring purposes. This means that some criteria which are clearly more important (eg, “defines method of allocation concealment”; Table 1⇑) than others (eg, title “identifies study as a RCT”) are not weighted to reflect their importance. Hence, some trials could in principal be better reported than others but have lower report quality scores. However, any weightings would be arbitrary and subjective and therefore equally subject to criticism. Hence, for simplicity we have left criteria unweighted. Second, several criteria are particularly subjective (eg, “interprets findings reasonably”), and we could have performed separate analyses including and excluding such criteria. However, because some of these subjective criteria were components of the CONSORT statement and therefore felt to be important by that group,13 we did not perform subgroup analyses of the CONSORT or extended criteria. Finally, the overall report quality score (based on 33 CONSORT criteria and 53 ad hoc criteria) was not tested for its properties (eg, internal consistency and construct validity), although interrater reliability was very good.
Randomization and Concealment of Allocation
We found that few trial reports adequately gave randomization details. Indeed, when we have approached trialists for precise details on randomization for an ongoing systematic review,19 several have been unable to define the exact method used. Previous studies have shown that nonrandomized trials yield larger estimates of treatment effect than those using randomized allocation20,21; hence, it is important that authors give detailed information on whether, and how, randomization was performed. It has been suggested previously that trialists keep examples of randomization material (eg, envelopes) so that others can ascertain how well randomization was performed and allocation concealed.
However, randomization does not prevent bias if investigators fail to prevent foreknowledge of treatment allocation. If trialists know what the next (randomly generated) treatment is (by being able to see through randomization envelopes, for example), they may enroll particular types of patients and thereby introduce bias. Trials with poor or unexplained concealment are more likely to yield larger estimates of treatment effect.14 21 Once again, we found that few trial reports gave satisfactory information on how allocation was concealed. A number of older trials used quasi-randomization methods, eg, date of birth, day of week, or alternation. Quasi-randomization methods are now unacceptable to most journals, because they can be associated with considerable bias due to a lack of concealment of allocation (eg, trialists can manipulate enrollment if they can predict which treatment the next patient will receive).22
Another area of concern where information was often absent in the trial reports was informed consent. Rikkert and colleagues23 found that only about half of RCTs gave information on informed consent and approval of research ethics committees. Although two thirds of acute stroke trials gave information on consent, less than one third mentioned approval of a research ecthics Committee.
Baseline Risk Factors
The presence (frequency) of prognostically important baseline clinical features should be reported; such factors include gender, stroke risk factors (atrial fibrillation; diabetes mellitus; and previous stroke, ischemic heart disease, and hypertension), stroke subtype (cortical, lacunar, and brain stem, for example, using the Oxford classification24). Likewise, prognostically important baseline “continuous” variables need reporting; these include age, blood pressure, neurological impairment, temperature, and glucose level.25 26 Poor baseline matching of patients for prognostic variables can affect whether a trial is positive, neutral, or negative, and it is important that trialists report these factors in full for each treatment group. Unfortunately, information was often missing on prognostic variables in the acute stroke trials.
Although the number of patients recruited is usually presented in trial publications, it is also important to report the number of subjects who were screened so that the relevance of the findings to the general population of stroke patients can be assessed. Detailed information is also required on the drug and placebo and their administration (route, frequency, delay in administration, and length of administration).
Our findings largely mirror those from other studies of RCT reports relating to publications in obstetric, pediatric, and general medicine journals8 9 23 27 28 29 or to specific conditions (eg, breast cancer and rheumatoid arthritis).30 31 A recurring theme is that information on randomization, sample size estimates, P values and confidence intervals, and consent are either insufficient or confusing. However, the absence of information in a report may not simply reflect poor writing; instead it often results from poor trial quality and the failure to perform relevant procedures.30
Journal editors should change instructions for authors to cover the reporting of RCTs to ensure that issues which affect the understanding of a paper and how the study was undertaken, whether by referee or general reader, are adequately described. Referees should then be asked to judge papers in this context. The CONSORT statements13 present guidelines to trialists, editors, and referees for improving the quality of reporting of RCTs.
We thank the Cochrane Stroke Review Group (Mrs H. Fraser, Edinburgh, UK) for providing a list of stroke trials. Dr F. Bath was supported by the NHS R&D Executive (South Thames; grant SPGS 236). Dr P. Bath was Wolfson Senior Lecturer in Stroke Medicine.
Presented in part at the 6th European Stroke Conference, Amsterdam, the Netherlands, May 29–31, 1997, and published in abstract form (Cerebrovasc Dis. 1997;7[suppl 4]:47).
- Received May 29, 1998.
- Revision received July 22, 1998.
- Accepted July 22, 1998.
- Copyright © 1998 by American Heart Association
Bath PMW. Treating acute ischaemic stroke. BMJ. 1995;311:139–140.
Wardlaw JM, Yamaguchi T, del Zoppo G. Thrombolytic therapy versus control in acute ischaemic stroke [Cochrane Review]. In: The Cochrane Library, Issue 2. Oxford, UK: Update Software; 1998. Updated quarterly.
Mosteller F, Gilbert JP, McPeek B. Reporting standards and research strategies for controlled trials. Control Clin Trials. 1980;1:37–58.
Bath P, Bath F. Prostacyclin and analogues in acute ischaemic stroke [Cochrane Review]. In: The Cochrane Library, Issue 2. Oxford, UK: Update Software; 1998. Updated quarterly.
Bath PMW, Bath FJ, Asplund K. Pentoxifylline, propentofylline and pentifylline in acute ischaemic stroke [Cochrane Review]. In: The Cochrane Library, Issue 2. Oxford, UK: Update Software; 1998. Updated quarterly.
Mohiuddin AA, Bath FJ, Bath PMW. Theophylline, aminophylline, caffeine and analogues, in acute ischaemic stroke [Cochrane Review]. In: The Cochrane Library, Issue 2. Oxford, UK: Update Software; 1998. Updated quarterly.
Siegel S, Castellan NJ. Nonparametric Statistics for the Behavioral Sciences. Singapore: McGraw-Hill Book Co; 1988:1–399.
Altman D. Better reporting of randomised controlled trials: the CONSORT statement. BMJ. 1996;313:570–571.
Bath FJ, Bath PMW. What is the correct management of blood pressure in acute stroke? The Blood Pressure in Acute Stroke Collaboration. Cerebrovasc Dis. 1997;7:205–213.
Altman D. Randomisation: essential for reducing bias. BMJ. 1991;302:1481–1482.
Olde Rikkert MGM, ten Have HAMJ, Hoefnagels WHL. Informed consent in biomedical studies on aging: survey of four journals. BMJ. 1996;313:1117.
Weir CJ, Murray GD, Dyker AG, Lees KR. Is hyperglycaemia an independent predictor of poor outcome after acute stroke? Results of a long-term follow up study. BMJ. 1997;314:1303–1306.
Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol. 1986;4:942–951.