Suppression of Clinical Trials of Sumatriptan

Correspondence to the Lancet two weeks ago revealed another instance in which clinical research studies that failed to provide results favorable to a sponsor's product were suppressed. [Tfeft-Hansen  PC. Unpublished clinical trials of sumatriptan. Lancet 2009; 374: 1501-2.  Link here.]

Sumatriptan, sold as Imitrex by GlaxoSmithKline, and now available generically, is a commonly used treatment for acute migraine headaches.  A Cochrane review from 2003 concluded that multiple clinical trials versus placebo showed that the drug is a safe and effective treatment of acute migraine [link here.]  A comparison of multiple guidelines for headache treatment noted considerable variability in how guidelines were developed, but that a number recommended sumatriptan as a first-line agent based apparently mainly on trials against placebo, while noting a lack of head-to-head comparisons among sumatriptan (and other triptans) and older, simpler treatments, like non-steroidal anti-inflammatory drugs (NSAIDs).  [Schuurmans A, van Weel C. Pharmacologic treatment of migraine: comparison of guidelines. Can Fam Physician 2005; 51: 838-843.  Link here.]

Tfelt-Hansen had written a review of treatment of migraine with a combination of ergotamine and caffeine, which included a single trial comparing that combination to sumatriptan.  He noted that the results of this trial appeared on "the homepage of the Swedish Medical Agency," but were not published in a peer-reviewed journal.  The trial suggested that sumatriptan was inferior to the combination. 

Tfelt-Hansen looked up this trial in the GlaxoSmithKline trial register.  Recall that this database was the source of information on unpublished trials of Avandia (rosiglitazone, GSK) that Dr Steven Nissen and colleagues meta-analyzed to suggest that Avandia may produce adverse cardiovascular effects (see post here).   Recall also that the registry was created as part of a settlement of a lawsuit by then New York Attorney General Elliot Spitzer that accused GSK of concealing clinical research unfavorable to its drug paroxetine (Paxil, GSK).  [Steinbrook R. Registration of clinical trials - voluntary or mandatory? N Engl J Med 2004; 351:1820-1822.  Link here.]

Tfelt-Hansen's main finding was that there were six trials comparing sumatriptan to other treatments, including paracetamol (acetaminophen) plus metaclopamide (2 trials), buclizine chloride, paracetamol and codeine (2 trials), ergotamine tartrate, cyclizine HCl, caffeine (1 trial), and ergotamine tartrate plus caffeine (1 trial).  In 3 trials, sumatriptan treated patients were not significantly more likely to have relief of their headaches within 2 hours.  In 1 trial, there was a non-significant trend favoring sumatriptan.  In two trials, sumatriptan was superior to the comparison.  In 4 trials, the rate of relief after sumatriptan treatment was 50% or less. 

Tfelt-Hansen concluded:

It is easy to understand why these RCTs were never published when sumatriptan was introduced: in only one of the oral trials did more than 50% of patients have headache relief (the primary efficacy measure) after sumatriptan 100 mg for the first attack treated (table), and in the RCT with rectal sumatriptan, the drug was found inferior to ergotamine. These findings would at the time have spoiled the very positive picture of sumatriptan as a new wonder drug for migraine.

Of course, this points out that health care corporations may regard clinical trials more as marketing tools than as science. We have discussed numerous instances in which trials that did not show commercial sponsors' products in a favorable light were suppressed by these sponsors.

Individual trials only at best produce approximations of the truth about the drugs or devices they compare. Trials may be positive due to chance alone when the test or treatment under study actually has no good effects. Post-hoc suppression of "negative" trials therefore may exaggerate the benefits (and safety) of tests or treatments. Physicians and patients who try to uphold the ideals of evidence-based medicine, and base decisions on the best possible evidence can be misled when evidence unfavorable to vested interests is systematically suppressed. Suppression of evidence unfavorable to vested interests may lead to excess use of tests and treatments that really are less beneficial or more risky than the published evidence suggested, and to willingness to pay exaggerated prices for such tests and treatments. Thus suppression of evidence can lead to excess costs and bad outcomes.

Also, as we have said before, suppression of results of clinical research that are unfavorable to the vested interests of research sponsors violates the trust of research subjects.  Research subjects are often assured that their participation is for the benefit of science and health care.  Suppressing results unfavorable to vested interests distorts science and makes health care more dysfunctional.

Evidence that is purposefully suppressed is by definition hard to find. Nonetheless, we have seen several recent examples in which suppressed evidence was later revealed, and when combined with existing evidence, showed that previously hyped treatments were really not as safe and effective as was thought.  In particular, the suppression by various drug companies of evidence unfavorable to new anti-depressants they were marketing has generated some discussion (see post here).

This parade of examples suggests that stronger measures are needed to assure that clinical research is not suppressed due to the vested interests of research sponsors.  One seemingly radical, but increasingly plausible approach would ban corporations that sell health care products or services from influencing clinical research done to evaluate those products.  Would be health care reformers who really want to improve outcomes, improve access and decrease costs might want to think about how to make the evidence available about the outcomes of health care interventions more honest.

Suppression of Clinical Trials of Sumatriptan

Correspondence to the Lancet two weeks ago revealed another instance in which clinical research studies that failed to provide results favorable to a sponsor's product were suppressed. [Tfeft-Hansen  PC. Unpublished clinical trials of sumatriptan. Lancet 2009; 374: 1501-2.  Link here.]

Sumatriptan, sold as Imitrex by GlaxoSmithKline, and now available generically, is a commonly used treatment for acute migraine headaches.  A Cochrane review from 2003 concluded that multiple clinical trials versus placebo showed that the drug is a safe and effective treatment of acute migraine [link here.]  A comparison of multiple guidelines for headache treatment noted considerable variability in how guidelines were developed, but that a number recommended sumatriptan as a first-line agent based apparently mainly on trials against placebo, while noting a lack of head-to-head comparisons among sumatriptan (and other triptans) and older, simpler treatments, like non-steroidal anti-inflammatory drugs (NSAIDs).  [Schuurmans A, van Weel C. Pharmacologic treatment of migraine: comparison of guidelines. Can Fam Physician 2005; 51: 838-843.  Link here.]

Tfelt-Hansen had written a review of treatment of migraine with a combination of ergotamine and caffeine, which included a single trial comparing that combination to sumatriptan.  He noted that the results of this trial appeared on "the homepage of the Swedish Medical Agency," but were not published in a peer-reviewed journal.  The trial suggested that sumatriptan was inferior to the combination. 

Tfelt-Hansen looked up this trial in the GlaxoSmithKline trial register.  Recall that this database was the source of information on unpublished trials of Avandia (rosiglitazone, GSK) that Dr Steven Nissen and colleagues meta-analyzed to suggest that Avandia may produce adverse cardiovascular effects (see post here).   Recall also that the registry was created as part of a settlement of a lawsuit by then New York Attorney General Elliot Spitzer that accused GSK of concealing clinical research unfavorable to its drug paroxetine (Paxil, GSK).  [Steinbrook R. Registration of clinical trials - voluntary or mandatory? N Engl J Med 2004; 351:1820-1822.  Link here.]

Tfelt-Hansen's main finding was that there were six trials comparing sumatriptan to other treatments, including paracetamol (acetaminophen) plus metaclopamide (2 trials), buclizine chloride, paracetamol and codeine (2 trials), ergotamine tartrate, cyclizine HCl, caffeine (1 trial), and ergotamine tartrate plus caffeine (1 trial).  In 3 trials, sumatriptan treated patients were not significantly more likely to have relief of their headaches within 2 hours.  In 1 trial, there was a non-significant trend favoring sumatriptan.  In two trials, sumatriptan was superior to the comparison.  In 4 trials, the rate of relief after sumatriptan treatment was 50% or less. 

Tfelt-Hansen concluded:

It is easy to understand why these RCTs were never published when sumatriptan was introduced: in only one of the oral trials did more than 50% of patients have headache relief (the primary efficacy measure) after sumatriptan 100 mg for the first attack treated (table), and in the RCT with rectal sumatriptan, the drug was found inferior to ergotamine. These findings would at the time have spoiled the very positive picture of sumatriptan as a new wonder drug for migraine.

Of course, this points out that health care corporations may regard clinical trials more as marketing tools than as science. We have discussed numerous instances in which trials that did not show commercial sponsors' products in a favorable light were suppressed by these sponsors.

Individual trials only at best produce approximations of the truth about the drugs or devices they compare. Trials may be positive due to chance alone when the test or treatment under study actually has no good effects. Post-hoc suppression of "negative" trials therefore may exaggerate the benefits (and safety) of tests or treatments. Physicians and patients who try to uphold the ideals of evidence-based medicine, and base decisions on the best possible evidence can be misled when evidence unfavorable to vested interests is systematically suppressed. Suppression of evidence unfavorable to vested interests may lead to excess use of tests and treatments that really are less beneficial or more risky than the published evidence suggested, and to willingness to pay exaggerated prices for such tests and treatments. Thus suppression of evidence can lead to excess costs and bad outcomes.

Also, as we have said before, suppression of results of clinical research that are unfavorable to the vested interests of research sponsors violates the trust of research subjects.  Research subjects are often assured that their participation is for the benefit of science and health care.  Suppressing results unfavorable to vested interests distorts science and makes health care more dysfunctional.

Evidence that is purposefully suppressed is by definition hard to find. Nonetheless, we have seen several recent examples in which suppressed evidence was later revealed, and when combined with existing evidence, showed that previously hyped treatments were really not as safe and effective as was thought.  In particular, the suppression by various drug companies of evidence unfavorable to new anti-depressants they were marketing has generated some discussion (see post here).

This parade of examples suggests that stronger measures are needed to assure that clinical research is not suppressed due to the vested interests of research sponsors.  One seemingly radical, but increasingly plausible approach would ban corporations that sell health care products or services from influencing clinical research done to evaluate those products.  Would be health care reformers who really want to improve outcomes, improve access and decrease costs might want to think about how to make the evidence available about the outcomes of health care interventions more honest.

Failing to Report Adverse Effects of Treatments

We have frequently advocated the evidence-based medicine (EBM) approach to improve the care of individual patients, and to improve health care quality at a reasonable cost for populations. Evidence-based medicine is not just medicine based on some sort of evidence. As Dr David Sackett, and colleagues wrote [Sackett DL, Rosenberg WM, Muir Gray JA, Haynes RB, Richardson WS. Evidence-based medicine; what it is and what it isn't. BMJ 1996; 312: 71-72. Link here. ]

Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.

One can find other definitions of EBM, but nearly all emphasize that the approach is designed to appropriately apply results from the best clinical research, critically reviewed, to the individual patient, taking into account that patient's clinical characteristics and personal values.

When making decisions about treatments for individual patients, the EBM approach suggests using the best available evidence about possible benefits and harms of treatment, so that the treatment chosen is most likely to maximize benefits and minimize harms for the individual patient. The better the evidence about specific benefits and harms applicable to a particular patient, the greater will be the likelihood that a particular decision based on this evidence will result in the best possible outcomes for the patient.

A new study in the Archives of Internal Medicine focused on how articles report adverse effects found by clinical trials. [Pitrou I, Boutron I, Ahmad N et al. Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med 2009; 169: 1756-1761. Link here.] The results were not encouraging.

The investigators assessed 133 articles reporting the results of randomized controlled trials published in 2006 in six English language journals with high impact factors, that is, the most prestigious journals, including the New England Journal of Medicine, Lancet, JAMA, British Medical Journal, and Annals of Internal Medicine. They excluded trials with less common designs, such as randomized cross-over trials. The majority of trials (54.9%) had private, or private mixed with public funding.

The major results were:

15/133 (11.3%) did not report anything about adverse events

36/133 (27.1%) did not report information about the severity of adverse events

63/133 (47.4%) did not report how many patients had to withdraw from the trial due to adverse events

43/133 (32.3%) had major limitations in how they reported adverse events, e.g., reporting only the most common events (even though most trials do not enroll enough patients to detect important but uncommon events).

The authors concluded, "the reporting of harm remains inadequate."

An accompanying editorial [Ioannidis JP. Adverse events in randomized controlled trials: neglected, distorted, and silenced. Arch Intern Med 2009; 169: 1737-1739. Link here] raised concerns about why the reporting of adverse events is so shoddy:

Perhaps conflicts of interest and marketing rather than science have shaped even the often accepted standard that randomized trials study primarily effectiveness, whereas information on harms from medical interventions can wait for case reports and nonrandomized studies. Nonrandomized data are very helpful, but they have limitations, and many harms will remain long undetected if we just wait for spontaneous reporting and other nonrandomized research to reveal them. In an environment where effectiveness benefits are small and shrinking, the randomized trials agenda may need to reprogram its whole mission, including its reporting, toward better understanding of harms.

Pitrou and colleagues have added to our knowledge about the drawbacks of the evidence about treatments that is publicly available to physicians and patients when making decisions about treatment. Even reports of studies with the best designs (randomized controlled trials) in the best journals seem to omit important information about the harms of the treatments they test.

It appears that the majority of the reports that Pitrou and colleagues studied received "private" funding, presumably meaning most were funded by drug, biotechnology, or device companies and were likely meant to evaluate the sponsoring companies' products. However, note that this article did not analyze the relationship of funding source to the completeness of information about adverse effects.

Nonetheless, on Health Care Renewal we have discussed many cases in which research has been manipulated in favor of the vested interests of research sponsors (funders), or in which research unfavorable to their interests has been suppressed. Therefore, it seems plausible that sponsors' influence over how clinical trials are designed, implemented, analyzed and reported may reduce information about the adverse effects of their products reported in journal articles. Trials may be designed not to gather information about adverse events. Analyses of some adverse events, or some aspects of these events may not be performed, or if performed, not reported. The evidence from clinical research available to make treatment decisions consequently may exaggerate the ratios of certain drugs' and devices' benefits to their harms.

Patients may thus receive treatments which are more likely to hurt than to help them, and populations of patients may be overtreated. Impressions that treatments are safer than they actually are may allow their manufacturers to overprice them, so health care costs may rise.

The article by Pitrou and colleagues adds to concerns that we physicians may too often really be practicing pseudo-evidence based medicine when we think we are practicing evidence-based medicine. We cannot judiciously balance benefits and harms of treatments to make the best decisions for patients when evidence about harms is hidden. Clearly, as Ioannidis wrote, we need to "reprogram." However, what we need to reprogram is our current dependence on drug and device manufacturers to pay for (and hence de facto run) evaluations of their own products. If health care reformers really want to improve quality while controlling costs, this is the sort of reform they need to start considering.

NB - See also the comments by Merrill Goozner in the GoozNews blog.

Failing to Report Adverse Effects of Treatments

We have frequently advocated the evidence-based medicine (EBM) approach to improve the care of individual patients, and to improve health care quality at a reasonable cost for populations. Evidence-based medicine is not just medicine based on some sort of evidence. As Dr David Sackett, and colleagues wrote [Sackett DL, Rosenberg WM, Muir Gray JA, Haynes RB, Richardson WS. Evidence-based medicine; what it is and what it isn't. BMJ 1996; 312: 71-72. Link here. ]

Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.

One can find other definitions of EBM, but nearly all emphasize that the approach is designed to appropriately apply results from the best clinical research, critically reviewed, to the individual patient, taking into account that patient's clinical characteristics and personal values.

When making decisions about treatments for individual patients, the EBM approach suggests using the best available evidence about possible benefits and harms of treatment, so that the treatment chosen is most likely to maximize benefits and minimize harms for the individual patient. The better the evidence about specific benefits and harms applicable to a particular patient, the greater will be the likelihood that a particular decision based on this evidence will result in the best possible outcomes for the patient.

A new study in the Archives of Internal Medicine focused on how articles report adverse effects found by clinical trials. [Pitrou I, Boutron I, Ahmad N et al. Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med 2009; 169: 1756-1761. Link here.] The results were not encouraging.

The investigators assessed 133 articles reporting the results of randomized controlled trials published in 2006 in six English language journals with high impact factors, that is, the most prestigious journals, including the New England Journal of Medicine, Lancet, JAMA, British Medical Journal, and Annals of Internal Medicine. They excluded trials with less common designs, such as randomized cross-over trials. The majority of trials (54.9%) had private, or private mixed with public funding.

The major results were:
15/133 (11.3%) did not report anything about adverse events
36/133 (27.1%) did not report information about the severity of adverse events
63/133 (47.4%) did not report how many patients had to withdraw from the trial due to adverse events
43/133 (32.3%) had major limitations in how they reported adverse events, e.g., reporting only the most common events (even though most trials do not enroll enough patients to detect important but uncommon events).

The authors concluded, "the reporting of harm remains inadequate."

An accompanying editorial [Ioannidis JP. Adverse events in randomized controlled trials: neglected, distorted, and silenced. Arch Intern Med 2009; 169: 1737-1739. Link here] raised concerns about why the reporting of adverse events is so shoddy:

Perhaps conflicts of interest and marketing rather than science have shaped even the often accepted standard that randomized trials study primarily effectiveness, whereas information on harms from medical interventions can wait for case reports and nonrandomized studies. Nonrandomized data are very helpful, but they have limitations, and many harms will remain long undetected if we just wait for spontaneous reporting and other nonrandomized research to reveal them. In an environment where effectiveness benefits are small and shrinking, the randomized trials agenda may need to reprogram its whole mission, including its reporting, toward better understanding of harms.

Pitrou and colleagues have added to our knowledge about the drawbacks of the evidence about treatments that is publicly available to physicians and patients when making decisions about treatment. Even reports of studies with the best designs (randomized controlled trials) in the best journals seem to omit important information about the harms of the treatments they test.

It appears that the majority of the reports that Pitrou and colleagues studied received "private" funding, presumably meaning most were funded by drug, biotechnology, or device companies and were likely meant to evaluate the sponsoring companies' products. However, note that this article did not analyze the relationship of funding source to the completeness of information about adverse effects.

Nonetheless, on Health Care Renewal we have discussed many cases in which research has been manipulated in favor of the vested interests of research sponsors (funders), or in which research unfavorable to their interests has been suppressed. Therefore, it seems plausible that sponsors' influence over how clinical trials are designed, implemented, analyzed and reported may reduce information about the adverse effects of their products reported in journal articles. Trials may be designed not to gather information about adverse events. Analyses of some adverse events, or some aspects of these events may not be performed, or if performed, not reported. The evidence from clinical research available to make treatment decisions consequently may exaggerate the ratios of certain drugs' and devices' benefits to their harms.

Patients may thus receive treatments which are more likely to hurt than to help them, and populations of patients may be overtreated. Impressions that treatments are safer than they actually are may allow their manufacturers to overprice them, so health care costs may rise.

The article by Pitrou and colleagues adds to concerns that we physicians may too often really be practicing pseudo-evidence based medicine when we think we are practicing evidence-based medicine. We cannot judiciously balance benefits and harms of treatments to make the best decisions for patients when evidence about harms is hidden. Clearly, as Ioannidis wrote, we need to "reprogram." However, what we need to reprogram is our current dependence on drug and device manufacturers to pay for (and hence de facto run) evaluations of their own products. If health care reformers really want to improve quality while controlling costs, this is the sort of reform they need to start considering.

NB - See also the comments by Merrill Goozner in the GoozNews blog.