Lies, Damn Lies, and…

Statistics don't lie. People do.

I have the greatest respect for statisticians, who methodically sift through messy data to determine what can confidently and honestly be said about them. But even the most sophisticated analysis depends on how the data were obtained. The miniscule false-positive rate for DNA tests, for example, is not going to protect you if the police swap the tissue samples.

One of the core principles in clinical trials is that researchers specify what they're looking for before they see the data. Another is that they don't get to keep trying until they get it right.

But that's just the sort of behavior that some drug companies have engaged in.

In the Pipeline informs us this week of a disturbing article in the New England Journal of Medicine. The authors analyzed twenty different trials conducted by Pfizer and Parke-Davis evaluating possible off-label (non-FDA-approved) uses for their epilepsy drug Neurontin (gabapentin).

If that name sounds familiar, it may be because Pfizer paid a $0.43 billion dollar fine in 2004 for illegally promoting just these off-label uses. As Melody Peterson reported for The New York Times and in her chilling book, "Our Daily Meds," company reps methodically "informed" doctors of unapproved uses, for example by giving them journal articles on company-funded studies. The law then allows the doctors to prescribe the drug for whatever they wish.

But the distortion doesn't stop with the marketing division.

The NEJM article draws on internal company documents that were discovered for the trial. Of 20 clinical trials, only 12 were published. Of these, eight reported a statistically significant outcome that was not the one that was described in the original experimental design. The authors say "…trials with findings that were not statistically significant (P≥0.05) for the protocol-defined primary outcome, according to the internal documents, either were not published in full or were published with a changed primary outcome."

A critical reason to specify the goals, or primary outcome, ahead of time is that the likelihood of getting a statistically significant result by chance increases as more possible outcomes are considered. In genome studies, for example, the criterion for significance is typically reduced by a factor that is the number of genes tested, or equivalently the number of possible outcomes.

None of this would be surprising to Peterson. She described a related practice in which drug companies keep doing trials until they get two positive outcomes, which is what the FDA requires for approval.

By arbitrary tradition, the numerical threshold for statistical significance is taken as a 5% or less chance that an outcome arose by chance (P-value). This means that if you do 20 trials you'll have a very good chance of getting one or more that are "significant," even if there is no effect.

A related issue arose for the recent, highly publicized results of an HIV/AIDS vaccine test in Thailand. Among three different analysis methods, one came up with a P-value of 4%, making it barely significant.

This means is that only one in twenty-five trials like this would get such a result by chance. That makes the trial a success, by the usual measures.

But this trial is just one of many trials for potential vaccines, most of which have shown no effect. The chances that any one of these trials gave a positive result is much larger, presumably more than 5%.

In addition, the Thai vaccine was expected to work by slowing down existing infection. Instead, the data show reduced rates of initial infection. Measured in terms of final outcome (death), it was a success. But in some sense the researchers moved the goalposts.

Sometimes, of course, a large trial can uncover a real but anticipated effect. It makes sense to follow up on these cases, recognizing that a single result is only a hint.

Because of the subtleties in defining the outcome of a complex study, there seems to be no substitute for repeating a trial, stating a clearly defined outcome. Good science writers understand this. It would be nice to think that the FDA did, too, and established procedures to ensure reliable conclusions.

Pharma in the News

Two big stories this week about how pharmaceutical industries get doctors to prescribe their drug:

• Some of the marketing plans of Forest Laboratories for their antidepressant Lexapro were made public. Their previous product, Celexa, which was approaching the end of its patent protection, contains a mixture of two mirror-image versions of the same molecule, while Lexapro contains only one. Generating a market for the newer, more expensive replacement, when it is so similar, takes a full-court "marketing" press. (See also this view from a pharmaceutical industry researcher at In the Pipeline.)
  
• For a record $2.3 Billion, Pfizer settles charges about their marketing practices. There have been several previous finds of many hundreds of millions of dollars, but apparently they judged the profit potential to be worth the risk.
  
• In a more encouraging report, the FDA says that 80% of a sample of postmarketing studies are proceeding on schedule. Since these studies include many more patients than those used for initial approval of drugs, they can uncover rare problems that were not statistically apparent in the original studies. Pharmaceutical companies have been criticized for dragging their feet on these studies, since they have little incentive to uncover new problems.

Where is the FDA?

In a previous post, I fretted that grand aspirations for personalized medicine don't gibe with drug companies' histories of doing whatever they can to expand sales, even to people who won't benefit. You would hope that this would not include clearly deceptive marketing, though, since in the U.S. this is supposed to be monitored for accuracy by the Food and Drug Administration.

You be the judge.

To set the stage, here's a current ad for Aricept, a drug that is approved and widely prescribed for Alzheimer's Disease:

Here's the critical quote:

"Studies showed Aricept slows the progression of Alzheimer's symptoms."

That statement has a clear meaning: if you take this drug you'll decline less than you otherwise would have. Of course this claim (which I've heard repeated by a doctor) is critical, because it implies that you're losing ground every month you don't take this drug, so you'd better get on it soon and stay on it (for the rest of your life). It is not the same thing as saying "Aricept relieves Alzheimer's symptoms," which would mean you could try it for a while and see if it helps.

Fortunately, those ultra-fine-print sheets that come with prescription drugs are now available online. Here's a pdf of the prescribing information for Aricept, from the Aricept.com website, which I'm confident was checked by the FDA. Go ahead and follow the link; it's tough reading, but a good habit, as you'll see.

At the top of the first page, in Figure 1, is a plot of the average cognitive scores over time, with and without the drug. (The shifts are small compared to the spread of individual responses, though, so many people who try it may not notice any effect). Here's the key: after patients are taken off the drug, their scores rapidly decline until they are indistinguishable from those of patients who never took it. But don't take my word for it, here's the conclusion in the fine print:

"This suggests that the beneficial effects of ARICEPT® abate over 6 weeks following discontinuation of treatment and do not represent a change in the underlying disease."

Am I missing something?

Who Will Personalize Medicine?

There's a lot of excitement in the biomedical community about the potential for "personalized medicine"--the tailoring of a patient's treatment to reflect his or her individual biology. The best known prospect is selecting between drug or dosing alternatives based on a DNA tests that may predict how a person will respond, but there are other ways to improve individual outcomes as well. There have been a few specific commercial drugs with genetic tests so far, notably erbitux and warfarin, with mixed results. Nonetheless, I have no doubt that the researchers are sincere in their hopes for improved treatment.

Unfortunately, the history of the pharmaceutical industry offers less basis for optimism. I've been reading the disturbing book Our Daily Meds, by former New York Times reporter Melody Peterson (Amazon, B&N). This is just one of several recent books documenting the cynical manipulation of the prescription-drug process by Big Pharma in service of their own profits. The list of manipulation techniques is long, including inventing new diseases to be treated by their newest drugs, evading requirements for full disclosure of side effects by advertising by hiring celebrity promoters and funding patient groups focused on individual diseases, flooding the scientific literature with ghost-written articles that favor their drug, encouraging prescriptions for off-label uses, creation and marketing of useless "me too" drugs, and much more.

The take-home message is that the pharmaceutical companies rarely limit their sales to patients who would truly benefit from them, which is what personalized medicine really requires. In fact, the companies have taken many opportunities to extend their drugs to diseases in which studies have shown little benefit, to downplay or deny side effects, and to open their markets to include new. unstudied populations. They also tolerate the fact that many recipients don't benefit from the drug they pay for.

Personalized medicine would demand the opposite, shrinking the market for each drug to those who actually respond. At the same time, the complexity and expense of clinical trials that subdivide the patients into subgroups will be much higher. It seems unrealistic to expect our current commercial and regulatory system to rise to this challenge without some major changes.