Friday, December 17, 2010

Are Most Published Studies Lies?

A couple of recent articles in the lay press expose the fallacy that publication in a scientific journal means "it must be true."  Thanks to Aunt Sue for noticing the first, published in the Atlantic by David Freedman, titled, "Lies, Damned Lies, and Medical Science," and the second published in the New Yorker by Jonah Lehnrer, titled, "The Truth Wears Off."  Both articles rely heavily on the work of John Ioannidis, an epidemiologist at Stanford.

Highlights (lowlights?) discussed in these articles:
  • Of the 49 most cited clinical research studies, only 34 have been replicated, and in 41% of cases where replication was attempted, the results contradicted the original findings or seriously downgraded the estimated effect size.
  • One third of all studies are never cited, let alone repeated
  • Of 432 genetic studies examined, most had serious flaws, and only 1 study held up when others attempted to replicate it
The public is slowly starting to catch on.  Spectacular failures such as Hormone Replacement Therapy, Cox-2 Inhibitors, and Vitamin E have demonstrated to the public that early results indicating that the benefits outweigh the burdens can collapse under closer scrutiny.  But while these studies were tainted by the corrosive influence of a profit motive, the problem is not limited to pharmaceutical sponsored trials.  It's pervasive throughout science.  Some explanations:
  • Regression to the mean.  Phenomena such as symptoms have natural variation; in other words, they wax and wane over time.  Patients are most likely to come to their doctors attention when they are most symptomatic.  If nothing were done, the most symptomatic patients would tend to feel better over time, just due to the natural variation of most symptoms.  But those same patients are enrolled in trials when they are most symptomatic, because that is when they come to their doctors attention.  They feel better over time and voila!  It must be the drug!  But had they done nothing, they likely would have felt better anyway.  An improvement in symptoms due to natural regression to the average level is mistakenly attributed to the drug: professors are promoted; drug companies make billions.
  • Publication bias.  Journals tend to favor publishing only positive studies.  In a recent paper in Archives of Internal Medicine by Emerson and colleagues, 210 reviewers were randomized to receive an article with a positive finding, or an identical article showing no difference.  97% recommended publication of the positive study, and only 80% recommended publication of the no-difference study.
  • Selective reporting.  Scientists find a way to confirm what they want to find.  Of 47 studies of acupuncture conducted in China, Taiwan, and Japan, 100% found acupuncture to be effective.  In contrast, only 56% of 94 studies of acupuncture conducted in the US, Sweden, and the UK were positive. 
So what are we to believe?  Are most published studies lies?  There is ample reason to be skeptical.  We need more support from journals and academics for replication studies.  And we shouldn't believe that just because something is published in a journal it is "the truth."

We should be skeptical of the following findings until we see repeated high quality evidence:
  • If a little vitamin D is good, a lot must be better
  • Opioids for non-cancer pain cause more suffering than benefit (see pallimed post)
  • New treatments for delaying and reversing Alzheimer's disease
  • Palliative care prolongs life
The closing words of the New Yorker article describe the conundrum:
Just because an idea is true doesn't mean it can be proved.  And just because an idea can be proved doesn't mean it's true.  When the experiments are done, we still have to chose what to believe.
by: Alex Smith

8 comments:

SB. Leavitt, MA, PhD said...

Bravo on your important blogpost above with links to some excellent resources. While I would argue that the medical research and publication establishments do not set out to lie intentionally, in most but not all cases, they are businesses and there is much at stake in the “publish or perish” environment. We have been trying via our ongoing series on Evidence-Based Pain Management at “Pain-Topics UPDATES” [here] to educate healthcare providers in becoming more skeptical and critical of everything they read and all of what they hear; albeit, we realize that most of those busy folks have relatively little time, training, or inclination in that regard.

Christopher Langston said...

Hey Alex - It is hard to tell if this is a really fundamental problem or just part of the self correcting work of science.

The drive to publish even drivel, the incentives towards submitting the "least publishable unit" so as to maximize the number of publications, and total disinterest in replication are all real issues.

Over the years, I've heard fairly serious proposals for things like: Publication review blind to results (to avoid bias). An institute for replication that would be supported by tithes from the field. And more stringent control on what research gets started on the grounds that bad research is "unethical."

But I think the most important tool is a good Bayesian perspective. You have to consider the prior odds in evaluating a study and unless a study's methodology is awfully good, no single study should overcome a negative prior probability.

Clearly there are some things where the level of public interest mixed with inadequately skeptical authors routinely creates over-stated/over-interpreted results. These would include the relationship of foods to health outcomes in epi studies (people want to believe) and the search for genes in family studies of disease (it just sounds so scientific).

In both these cases, the popular and naive theory of causality (what you eat can make you healthy or sick) and single "genes" are the major causes of disease are just wrong and the prior probabilities should be adjusted against those interpretations.

Alex Smith said...

Thanks SB Leavitt and Chris Langston for your comments! To the first, I agree most people do not set out to lie. In many cases, the presentation of erroneous findings stems from an unconscious desire to find something exciting. "False" would be a less judgmental word for these cases.

Chris - enjoyed the idea of a Bayesian approach to reading and reviewing research. And also completely agree with your examples of foods and genes: hope does funny things to people's prior odds.

sei said...

I'm troubled by the closing words of the article and your post. My reactions:

1) Just because an idea is true doesn't mean it can be proved. TRUE

2) And just because an idea can be proved doesn't mean it's true. TRUE, WITH MAJOR QUALIFICATIONS

3) When the experiments are done, we still have to choose what to believe. NO, NO, NO!

Starting with #2, all studies have their limitations. Thus, an idea can be proved in a specific study w/specific limitations. Within this context, the experiment may "prove" and idea. However, you extend the "proof" to other contexts at your peril. Thus, I'd re-phrase 2:

And just because an idea can be proven in one setting doesn't mean it's true in all settings.

Of course, one reason why limitations are often ignored is the mass media interest in attention-grabbing headlines, but that's another post.

Number 3 strikes me as overly nihilistic and anti-science. To say that one's belief on a subject is completely separate from the experimental results leads us down a path where the only explanation for natural phenomena is faith and fable. I'd argue that this distrust of science is what leads so many Americans to believe that creationism is equally valid as evolution (scientifically) or deny climate change.

Disclaimer: As a clinician-researcher, I have a personal interest in advancing and elevating science.

Alex Smith said...

Thanks Sei, quick response: don't we also decide what to believe after reading through the literature? I think this is what we do at the end of journal club, when we discuss how this article will affect our practice. Would we now test our patients for early alzheimer's dementia? Would we refer older adults with aortic stensosis who are non-surgical candidates for a trans-arterial valve? We take a vote. People still disagree, because we interpret the evidence differently, and at the end of the day, our beliefs and hopes are still mixed in with our scientific understanding.

On the other hand, I completely understand your concerns about discrediting science altogether.

Helen Chen said...

Let's not forget our ethical responsibility to 'do no harm'. We do need to continually question whether the "latest scientific evidence" is (or remains, over time)true, meaningful, and applicable to our patients. Remember that the 1949 Nobel prize was awarded for the 'breakthrough' in managing psychotic illness--leucotomy, AKA a form of lobotomy.

Dan Matlock said...

Alex, I agree with you that when the experiements are done, we still have to decide what to believe. Indeed, I always read the "conflicts of interest" even in the most robustly designed RCT (or, especially in the most robustly designed RCTs).

I think there really is no such thing as objective research. By the simple fact that you ask a question, you are revealing your biases. Example: If you are asking the question that functional status is a more powerful predictor of adverse outcomes than co-morbidity is, then you are already biased in that direction.

I think the best science we will ever be able to do is when we are trualy honest with ourselves regarding what our biases are. I included my "bias" towards patient-centered care in the limitation section of a qualitative study I just submitted as it almost assuredly influenced the themes I gleaned from the data.

Rogue Medic said...

The articles mentioned are written for the non-scientific community. The articles are written to sell advertising. The articles are biased.

That does not mean that there are not significant problems with medical research. We need to examine the study methodology carefully and look for any evidence of failure to control for variables. Too many studies do a poor job of this.

There are some good reviews of these articles at Science-Based Medicine.

Lies, damned lies, and…science-based medicine?

http://www.sciencebasedmedicine.org/?p=7734

and -

The “decline effect”: Is it a real decline or just science correcting itself?

http://www.sciencebasedmedicine.org/?p=8987

The example of regression to the mean is exactly why studies should have placebo controls, where practical.

The recent research on acupuncture that has included placebo acupuncture (sham acupuncture) showed that regular acupuncture did not do as well as sham acupuncture. Even the specially tailored acupuncture, that is supposed to be better than regular acupuncture, was not as good as placebo acupuncture. The acupuncturists, the gullible press, and even some medical reviewers were trying to figure out what was the special property of qi that the sham acupuncture was able to use to produce such positive results. They all ignored the fact that when the placebo is better than the treatment, that means the treatment does not work.

If a third of studies are not replicated or cited, what does that matter? That means that they are not read and do not influence anything except to pad the CV of the authors. If a study is printed in a journal and nobody reads it, does it have an effect? No.