doi: 10.1161/01.STR.0000024110.82735.5A
(Stroke. 2002;33:2043.)
© 2002 American Heart Association, Inc.
Original Contributions |
Study Design and Outcome Measures in Studies on Aneurysmal Subarachnoid Hemorrhage
From the Department of Neurology (I.C. van der S., Y.M.R., G.J.E.R., J. van G.) and Julius Center for General Practice and Patient Oriented Research (A.A.), University Medical Center Utrecht, Utrecht, Netherlands.
Correspondence to G.J.E. Rinkel, MD, University Department of Neurology, PO Box 85500, 3508 GA Utrecht, Netherlands. E-mail G.J.E.Rinkel{at}neuro.azu.nl
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Background and Purpose— Methods of performing and reporting randomized clinical trials (RCTs) are available, but weaknesses still occur. For observational studies, methodology is less well described, and weaknesses are even more likely. In recent guidelines for patients with subarachnoid hemorrhage (SAH), 25% of treatment recommendations are based on clinical trials. To interpret the results of research on the therapeutic effect of treatment modalities, definition of outcome measures is essential. We assessed quality of study design and outcome measures and presence and precision of definitions concerning major complications of SAH in studies evaluating treatment strategies in patients with aneurysmal SAH.
Methods— We retrieved and reviewed all articles on treatment strategies in patients with SAH that fulfilled a prespecified set of criteria and were published during 1990–1999 in 10 general, neurosurgical, or neurological journals. We categorized articles into RCTs, observational studies with a control group, and observational studies without a control group. We assessed study design by means of a prespecified set of methodological criteria. For outcome measures we assessed whether a prespecified outcome measurement was defined and whether any handicap scale was used. For complications after SAH we assessed whether the definition included a description of the clinical features and a technical investigation with criteria for abnormal results.
Results— We identified 18 RCTs, 24 observational studies with a control group, and 41 uncontrolled observational studies. Two RCTs, no observational studies with a control group, and 15 observational studies without a control group met all criteria for study design. A primary outcome measure was specified in 67 of the 83 studies and was defined in 59. Any measure of handicap or information on degree of dependence was given in 65 of 83 studies. A complete definition of delayed cerebral ischemia was given in 13 of 66 studies, of rebleeding in 2 of 26 studies, and of hydrocephalus in 2 of 14 studies.
Conclusions— Most studies on treatment strategies in SAH suffer from methodological weaknesses. This implies that current management of patients with SAH is based on weak evidence.
Key Words: clinical trials • outcome assessment • subarachnoid hemorrhage
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The methodology of randomized clinical trials (RCTs) has been studied extensively, and guidelines on performing and reporting of RCTs are available,1,2 but RCTs often have methodological weaknesses.3–5 Such methods are less well formalized for performing and reporting observational studies, and these studies may therefore be weaker. In the guidelines of the Stroke Council of the American Heart Association, approximately 25% of treatment recommendations for patients with aneurysmal subarachnoid hemorrhage (SAH) were based on evidence of 1 or more RCTs. The remaining 75% of recommendations for the treatment of patients with SAH resulted from observational studies.6 For the implementation of treatment strategies, not only the type of study (RCT or observational study) but also the methodological content should be considered.
In reports of research on treatment strategies, an accurate definition of outcome measures is essential. An important prognostic factor for a poor outcome in patients with SAH is the neurological condition of the patients after the initial hemorrhage,7 but the subsequent complications of rebleeding, delayed cerebral ischemia (DCI), and hydrocephalus often influence the eventual outcome of the patients as well.
We reviewed articles about treatment in patients with aneurysmal SAH published in leading journals during 1990–1999. We assessed (1) overall quality of study design, (2) outcome measures, and (3) presence and precision of definitions of the 3 major complications of SAH, ie, rebleeding, ischemia, and hydrocephalus.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Search Strategy
We identified articles during 1990–1999 concerning patients with aneurysmal SAH in the following 10 journals: New England Journal of Medicine; Lancet; BMJ; Journal of the American Medical Association; Annals of Neurology; Neurology; Journal of Neurology, Neurosurgery and Psychiatry; Stroke; Journal of Neurosurgery; and Neurosurgery. We accomplished this by hand-searching, via the annual indexes if available, for the following key words: subarachnoid bleeding/hemorrhage, aneurysm, intracranial aneurysm.
Eligibility
Articles were included on the basis of the following criteria: (1) report on a study about a certain treatment strategy, (2) full article (no abstract or letter), (3) direct observation of patients (no review or meta-analysis), and (4) number of patients 
5. Exclusion criteria were as follows: (1) combined report on patients with and without SAH and (2) report on patients with SAH from causes other than a ruptured aneurysm. We categorized articles according to study design into (1) RCTs, (2) observational studies with a control group, and (3) observational studies without a control group. We used separate criteria for each study design.
Definitions
Criteria for study design and definitions of outcome measures were derived from 2 standard textbooks8,9 and from guidelines for reporting on RCTs.1 Criteria for the definition of the 3 major complications from SAH were based on a previous and similar study on the precision of definitions of complications of aneurysmal SAH during 1985–1992.10
For the study design of RCTs, we assessed whether (1) the method of randomization was reported, (2) all patients who entered the study were accounted for in the analysis, (3) patients were analyzed in the treatment groups to which they were randomized, (4) blinded outcome assessment was used, (5) information on power calculation was given, (6) equality of treatment in the different trial arms (aside from the experimental treatment) was ensured and accounted for, and (7) baseline characteristics of treatment groups were balanced at the start of the trial, and, in case of imbalance, adjusted for in the analyses.
For observational studies with a control group, we assessed whether patients and controls were stratified for the following major prognostic factors: (1) clinical condition on admission, (2) delay between SAH and admission, (3) severity of hemorrhage on CT scan, and (4) site and size of the ruptured aneurysm. If patients and controls were not stratified for these prognostic factors and the distribution was unbalanced, we assessed whether the factors were adjusted for.
Criteria for study design on observational studies without a control group were as follows: (1) the study population should be a consecutive cohort of patients, (2) included patients should fulfill prespecified criteria, and (3) information on study design (retrospective or prospective) should be given.
For the outcome measures in all categories of studies, we assessed (1) whether a prespecified outcome measure was defined and (2) whether any scale of handicap was used or information on degree of dependence was given.
Furthermore, we assessed whether the 3 major complications from SAH (rebleeding, DCI, and hydrocephalus) were precisely defined if studied. We considered the definition of each of these 3 major complications adequate if it consisted of any description of required clinical features, supplemented by technical investigation with specification of criteria for abnormal results.
Data Extraction
Two observers (I.C. van der S. and Y.M.R.) independently assessed eligibility of studies and extracted data from the included studies, using the aforementioned criteria on study design and definitions on outcome measures and major complications. If no information on a criterion could be found, we categorized this criterion as absent. For observational studies with a control group, we additionally used the category "not applicable" in case a criterion could not be applied. For example, if a study compares the proportion of surgical complications between patients in a good and a poor clinical condition before operation, stratification for the criterion "clinical condition on admission" is not useful. In case of disagreement between the 2 adjudicators, consensus was reached by reviewing the study together. If disagreement remained, the article was discussed with all authors until agreement was achieved. We set out to conduct a separate analysis for articles in neurological/neurosurgical journals and those in general medical journals and a separate analysis according to the impact factor of the journals. Furthermore, we compared the presence and precision of definitions of outcome measures and of definitions of the 3 major complications between RCTs and observational studies.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
We identified 83 articles that fulfilled the inclusion and exclusion criteria of this study; these articles reported on 20 020 patients. Of these 83 articles, 71 were published in neurosurgical journals, 10 in neurological journals, and 2 in general journals. Because we could only identify 2 articles on the treatment of SAH in the general journals (both published in Lancet), we did not perform a separate analysis for neurological/neurosurgical versus general journals. Since 86% of articles were derived from only 2 journals (Journal of Neurosurgery and Neurosurgery), an analysis according to the impact factor of the article was also not useful. Eighteen studies were RCTs (5995 patients), 24 studies were observational with the use of a control group (4458 patients), and 41 studies were observational without the use of a control group (9567 patients).
Study Design
Figure 1 illustrates the number of RCTs fitting the criteria for study design. Two of the 18 RCTs met all 7 criteria; these 2 studies reported on 203 patients (3.4% of all patients included in the 18 RCTs). The median number of criteria that were met for study design of RCTs was 5.5.
|
Information on criteria for study design of observational studies with a control group is shown in Figure 2. None of these 24 studies met all 4 criteria for study design. The median number of criteria that were met for study design of observational studies with a control group was 2.
|
Figure 3 shows data on study design for observational studies without a control group. Fourteen (34%) of the 41 observational studies without a control group were prospective, 13 studies (32%) were retrospective, and in 14 studies (34%) no information on the study design was given. Fifteen studies met all 3 criteria for study design; these reported on 8109 patients (85% of all patients included in observational studies without a control group). The median number of criteria that were met for study design of observational studies without a control group was 2.
|
Definition of Outcome Measures
Sixty-seven of all 83 studies had a prespecified primary outcome measure, which was defined in 59 studies (88%). The primary outcome measure consisted of a scale on handicap or information on degree of dependence in 16 of 67 studies (24%). In 65 of the 83 studies (78%), a scale on handicap or information on degree of dependence was used for at least 1 of the outcome measures. The median number of fulfilled criteria for outcome measures was 2. In 14 of the 18 RCTs a prespecified primary outcome was used, which was defined in 13 studies. Of the 65 observational studies, a prespecified primary outcome measure was used in 53 studies, which was defined in 46 studies. In 16 RCTs and in 49 observational studies, at least 1 clinically relevant outcome measure was used.
Definition of Complications
Of the 83 studies included, 66 (80%) reported on DCI, 26 studies (31%) on rebleeding, and 14 studies (17%) on hydrocephalus. Eight studies (10%) reported on all 3 specific complications, 17 studies (20%) on 2, 48 studies (58%) on a single complication, and 10 studies (12%) on none of the 3 complications. The proportions of articles fulfilling the criteria for complete definition of the 3 major complications are shown in Figure 4. A complete definition of DCI was given in 13 of the 66 studies (20%) that used DCI as an outcome event (corresponding to 3676 patients, 20% of the patients for whom DCI was used as an outcome measure). The median number of criteria fulfilled for the definition of DCI was 2. Rebleeding was completely defined in 2 of the 26 studies (8%), corresponding to 489 patients (4.5%). The median number of criteria that were met was zero. A complete definition of hydrocephalus was given in 2 of the 14 studies (14%) (corresponding to 489 patients, 4.5% of the patients in whom occurrence of hydrocephalus was recorded). The median number of criteria fulfilled to define hydrocephalus was 0.5. Sixteen RCTs reported on DCI, and a complete definition was given in 7 of them. Of the 53 observational studies reporting on DCI, a complete definition was given in 6 studies. In 2 RCTs rebleeding was used as an outcome measure, and in both studies it was not completely defined. Of the 24 observational studies reporting on rebleeding, 2 studies had an adequate definition of this outcome measure. Two RCTs reported on hydrocephalus, and in both the criteria for complete definition were not fulfilled; 12 observational studies reported on this outcome measure, and a complete definition was given in 2 of them.
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The majority of studies reporting on treatment strategies in SAH do not fulfill generally accepted criteria for good methods. This applies to RCTs as well as to observational studies with or without a control group. The major complications after SAH were adequately defined in <20% of studies, and in 25% of the studies that mentioned outcome, no degree of handicap or dependence was given. There was no marked difference in presence and precision of definition of outcome measures and of definitions of the 3 major complications between RCTs and observational studies.
For evidence on treatment strategies, not only the type of study (RCT or observational study) but also the methodological content of a study should be considered. Few recommendations for treatment of SAH are based on RCTs; most are based on observational studies.1 The implication of the poor methodological quality of these trials and observational studies is that treatment strategies are based on weak evidence.
We used more stringent requirements for reporting on study design for RCTs than for observational studies because guidelines on methodological requirements are usually based on the former study type. For this reason, we did not compare the proportion of RCTs fulfilling the criteria for study design with the proportion of observational studies fulfilling these criteria.
Our results are an underestimation of the methodological problems in studies on SAH. We selected studies from prominent journals, with high impact factors.11 The studies we selected already met the standards of these journals. Therefore, any selection bias in the reviewed literature is likely to be in the direction of good methods.
Our analysis was limited to the question of whether specific criteria for study design and complications were mentioned. Some studies may not have mentioned adequate criteria but still may have used them. Conversely, an adequate description of criteria does not necessarily imply that these criteria are adequately applied.5
In 1994, a similar survey assessed the precision of definitions used for reporting on initial grade; the specific complications of rebleeding, DCI, and hydrocephalus; and the overall outcome in patients with aneurysmal SAH during 1985–1992.10 In that report, two thirds of included studies had incomplete or even no criteria for defining complications. In our study criteria for precise definition of complications were more stringent.
We conclude that most studies on treatment strategies in SAH still suffer from many methodological weaknesses. The implication of this finding is that most advice and guidelines on treatment strategies are based on weak evidence. Because guidelines for performing and reporting are available for both RCTs and observational studies, resources and patients should be directed only toward well-designed studies.
|
Acknowledgments |
|---|
This study was supported in part by an established clinical investigator grant from the Netherlands Heart foundation to Dr Rinkel (grant D98.014). The authors will provide the complete list of articles reviewed in this study on request.
Received November 5, 2001; revision received March 8, 2002; accepted April 29, 2002.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
- Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996; 276: 637–639.[CrossRef][Medline] [Order article via Infotrieve]
- Moher D, Schulze KF, Altman D, for the CONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. JAMA. 2001; 285: 1987–1991.
[Abstract/Free Full Text] - Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995; 273: 408–412.[Abstract]
- Hollis S, Campbell F. What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ. 1999; 319: 670–674.
[Abstract/Free Full Text] - Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ. 1998; 317: 1185–1190.
[Abstract/Free Full Text] - Mayberg MR, Batjer HH, Dacey R, Diringer M, Haley EC, Heros RC, Sternau LL, Torner J, Adams HP Jr, Feinberg W, Thies W. Guidelines for the management of aneurysmal subarachnoid hemorrhage: a statement for healthcare professionals from a special writing group of the Stroke Council, American Heart Association. Stroke. 1994; 25: 2315–2328.[Medline] [Order article via Infotrieve]
- Hijdra A, van Gijn J, Nagelkerke NJ, Vermeulen M, van Crevel H. Prediction of delayed cerebral ischemia, rebleeding, and outcome after aneurysmal subarachnoid hemorrhage. Stroke. 1988; 19: 1250–1256.
[Abstract/Free Full Text] - Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine: How to Practice and Teach EBM. 2nd ed. Edinburgh, Scotland: Churchill Livingstone; 2000.
- Fletcher RH, Fletcher SW, Wagner EH. Clinical Epidemiology: the Essentials. 3rd ed. Baltimore, Md: Williams & Wilkins; 1996.
- van Gijn J, Bromberg JE, Lindsay KW, Hasan D, Vermeulen M. Definition of initial grading, specific events, and overall outcome in patients with aneurysmal subarachnoid hemorrhage: a survey. Stroke. 1994; 25: 1623–1627.[Abstract]
- Institute for Scientific Information. Science Citation Index: Journal Citation Reports. Philadelphia, Pa: ISI Press; 1991.
This article has been cited by other articles:
 |
|
 |
|
  A Petzold, G Keir, A Kay, M Kerr, and E J Thompson Axonal damage and outcome in subarachnoid haemorrhage. J. Neurol. Neurosurg. Psychiatry, June 1, 2006; 77(6): 753 - 759. [Abstract] [Full Text] [PDF]
|
|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||