Response to Letter Regarding Article, “Guidelines for the Prevention of Stroke in Patients With Stroke and Transient Ischemic Attack: A Guideline for Healthcare Professionals From the American Heart Association/American Stroke Association”
We think letter of Dr Saver et al raises 3 important questions. First, what is the evidence required for a class III no benefit classification? Second, does the current evidence for the effectiveness of patent foramen ovale (PFO) closure devices meet the criterion for class III no benefit. Third, what are the appropriate criteria for assigning a level of evidence (LOE) A rating for the class III no benefit classification?
In assigning recommendation classes, the writing group was guided by the 2010 American Heart Association (AHA) methodology manual and policies from the American College of Cardiology Foundation/AHA Task Force on Practice Guidelines that includes the following advice:
Class III is reserved for conditions for which there is evidence or general agreement that the procedure/treatment is not useful/effective and in some cases may be harmful.
a. No benefit—procedure/test not helpful or treatment without established proven benefit
b. Harm—procedure/test leads to excess cost without benefit or is harmful, and or treatment is harmful.
Any combination of classification of recommendation and level of evidence is possible. For example, a recommendation can be a class I, even if it is based entirely on expert opinion and no research studies have ever been conducted on the recommendation (level C). Similarly, a class IIa or IIb can be assigned a level A if there are multiple randomized controlled trials coming to divergent conclusions.
The AHA does not provide detailed guidance on how to interpret medical literature to arrive at a designation of class III—no benefit. A recent article on the evolution and future of AHA guidelines (published after the stroke secondary prevention guideline) simply states “Class III: No Benefit is a moderate recommendation that is applied infrequently when evidence suggests that a strategy is no better than the control.”1 In common use, however, a new procedure is often deemed to be helpful, useful, and of proven benefit once it is shown to produce a clinically and statistically significantly better outcome compared with the standard therapy. A procedure is often deemed unhelpful, not useful, or of unproven benefit when this condition is not met. The writing group adopted the latter position in concluding that evidence from the 3 PFO superiority trials was enough to conclude that PFO closure was of unproven benefit. Dr Saver et al criticize the writing group for adopting this position, asserting that the writing group seems to have succumbed to the common fallacy of interpreting a superiority trial that is formally nonpositive as demonstrating neutrality. The writing group, however, has made no such assertion of neutrality. Instead, the writing group concluded that the 3 trials have not proven benefit for PFO closure.2
Dr Saver et al argue that, despite no proven benefit for PFO closure, our recommendation should be IIb because the point estimates for the hazard ratios from all 3 trials favor the new treatment. We acknowledge that, on this basis, class IIb might be justified. The writing group was aware of this option when it chose not to assign IIb based partially on the facts that harms had been identified in the unblinded trials (eg, procedure-related complications and atrial fibrillation) and studies had not documented the long-term risk:benefit ratio.3 A class IIb recommendation indicates that benefit≥risk; the writing group determined that class III no benefit more accurately reflected its interpretation of the evidence.
Dr Saver suggests that the writing group might have prespecified an equivalence margin and then applied this to a pooled analysis of the superiority trials. Such an analysis would go far beyond the usual expectations for a writing group. In addition, the meaning of findings from such an analysis would be uncertain. Equivalence and noninferiority trials require a specific active control, for which there is none for PFO among patients with cryptogenic stroke.
It has been argued that a class III designation could have the unwanted effect of discouraging use of PFO closure in patients who might truly benefit. Although research has identified patients who may be most likely to benefit from PFO closure, this effort has not yet reached fruition with independent validation.4–7 The committee recognizes the principle that guidelines should be mindful of unintended consequences. In the case of PFO closure, however, there are competing unintended consequences to consider. Class IIb designation could result in the use of PFO closure devices in a broad range of patients before the devices are proven to be clinically beneficial, whereas class III might constrain use in more selected patients who might benefit. Ultimately, the writing group made its decision for class III no benefit based on its interpretation of the available evidence. As is the case for all guidelines, clinicians will need to decide if this evidence applies to their specific patients and, therefore, if PFO closure should be considered even in the absence of more complete data. The writing group understands that the available trials on PFO closure do not provide adequate answers about the potential effectiveness of this procedure in all groups and under all circumstances.
It has also been argued that the class III designation for use of PFO closure devices might impair recruitment into ongoing or planned randomized clinical trials of PFO closure devices. The writing group firmly thinks that further data from randomized clinical trials are warranted and would be dismayed if the AHA guidelines had the effect of hindering enrollment. We are confident, however, that researchers can explain to patients the state of the evidence and the meaning of a class III no benefit designation. Devices and other therapies with potential but unproven benefit are precisely the devices and therapies that should be studied in clinical trials. If future trials confirm a benefit for PFO closure, the AHA has an established mechanism for issuing a scientific update.
The third important question raised by Dr Saver et al has to do with the level of evidence required for class III no benefit designation. The AHA has written that, class III: no benefit recommendations should not be associated with weak evidence or expert opinion—LOE C or E—because it is virtually impossible to ascertain lack of benefit without randomized trials or carefully conducted observational studies when safety is not the primary concern.1 However, the AHA does not define weak or strong evidence.
Dr Saver et al take a position that is even more restrictive than the AHA’s. They argue that the class III no benefit designation should be reserved for cases in which equivalence trials fail to show equivalence or noninferiority trials fail to show noninferiority. Because equivalence and noninferiority trials are still not common, a criterion that required evidence from these trials might not be used often.
In assigning a LOE of A to PFO closure device therapy, the writing group gave strong consideration to 3 published randomized clinical trials, 1 involving the STARFlex device8 and 2 involving the Amplatzer PFO Occluder.9,10 Each of the 3 were designed as superiority trials and all were open-label. A variety of antithrombotic regimens were permitted in the comparison groups. None of the trials found a statistically significant benefit for the primary outcome. Of note, average annual stroke outcome rates in the comparison groups were uniformly low (≈0.6% to 1.6% per year). Procedural complications were observed for 1.5% to 3.2% of patients treated with PFO closure devices. A meta-analysis of the 3 trials examined the effect of PFO closure for the outcome of recurrent stroke.11 The findings showed a low overall risk for stroke in both the device groups (0.76 events per 100 person-years) and medical therapy groups (1.3 per 100 person-years). Closure was associated with a 45% reduction in stroke risk that did not reach statistical significance (hazard ratio, 0.55; 95% confidence interval, 0.26–1.18). When the author looked at just the trials using the Amplatzer device, the hazard ratio remained nonsignificant. In reviewing the trials and the meta-analysis, the writing group concluded that the trials supported an LOE assignment of A. The criteria for LOE A from the AHA is multiple populations evaluated. “Data derived from multiple randomized clinical trials or meta-analyses.”1,2 In our opinion, PFO closure devices met this criteria.
In summary, we respectfully disagree with assertion of Dr Saver et al that the writing group made an error in class assignment for use of PFO closure devices, and we do not think any change to PFO recommendation number 4 is warranted. The assertion that the AHA/ACC/ASA evidentiary process requires a class IIb rather than a class III recommendation is difficult to square with the guidance statement we reproduce above. In the judgment of the writing group, the effectiveness of PFO closure is unproven. We acknowledge, however, that the distinction between classes IIb and III designation for PFO closure is a matter of reasonable judgment and that the AHA guidance document can be cited to support either classification. Class IIb can be supported by claiming that the 3 PFO trials leave uncertain the effectiveness of PFO closure (and keep the possibility of benefit alive). Class III can be supported by claiming that benefit has not been proven. We voted for class III. Contrary to the assertion of Dr Saver et al, the writing group made no claim of definitely no benefit for PFO closure. The wording of our recommendation is as follows, “For patients with a cryptogenic ischemic stroke or TIA [transient ischemic attack] and a PFO without evidence for DVT [deep venous thrombosis], available data do not support a benefit for PFO closure (Class III; Level of Evidence A).”2 Contrary to assertion of Dr Saver et al, the committee has not interpreted the evidence as showing no benefit, just no proven benefit.
The writing group agrees with the authors of the meta-analysis cited above who concluded that further randomized data and longer follow-up from completed trials, if available, are needed and should be obtained to resolve uncertainty about the potential benefits and risks for PFO closure. It is our hope and expectation that the class III no benefit designation, as discussed above, will not impair the conduct of adequately powered trials designed to identify a statistically significant benefit for PFO closure in carefully selected patients.
To avoid controversies similar to this one in the future, we recommend that all interested parties work with the AHA to issue more detailed guidance about the application of the class III no benefit recommendation, with attention to evaluations of LOE. We think it may be helpful to adopt wording that either (1) further clarified the distinctions between harm, no benefit, and no proven benefit in the class III designation or (2) further explains the broad range covered in this designation.
The AHA guideline protocol is a document in constant evolution and is designed to accommodate future improvements.1
This reply was read and approved by the 3 authors. In addition, the letter was referred to the entire writing group for approval. A majority of the group, with the exception of those who recused themselves, approved.
Walter N. Kernan, MD
Yale University School of Medicine
New Haven, CT
Bruce Ovbiagele, MD, MSc, MAS
Medical University of South Carolina
Steven J. Kittner, MD, MPH
University of Maryland School of Medicine
for the Secondary Prevention Guideline Writing Group
Stroke welcomes Letters to the Editor and will publish them, if suitable, as space permits. Letters must reference a Stroke published-ahead-of-print article or an article printed within the past 3 weeks. The maximum length is 750 words including no more than 5 references and 3 authors. Please submit letters typed double-spaced. Letters may be shortened or edited.
The American Heart Association/American Stroke Association Manuscript Oversight Committee reviewed this letter on October 20, 2014.
Guest Editor for this article was Bo Norrving, MD, PhD.
- © 2015 American Heart Association, Inc.
- Jacobs AK,
- Anderson JL,
- Halperin JL
- Kernan WN,
- Ovbiagele B,
- Black HR,
- Bravata DM,
- Chimowitz MI,
- Ezekowitz MD,
- et al
- Udell JA,
- Opotowsky AR,
- Khairy P,
- Silversides CK,
- Gladstone DJ,
- O’Gara PT,
- et al
- Thaler DE,
- Ruthazer R,
- Di Angelantonio E,
- Di Tullio MR,
- Donovan JS,
- Elkind MS,
- et al
- Kitsios GD,
- Thaler DE,
- Kent DM