Dirty Little Secret

/ by João Medeiros /

Are most published research findings actually false? The case for reform.

Illustration by Jeffrey Docherty

In a 2005 article in the Journal of the American Medical Association, epidemiologist John Ioannidis showed that among the 45 most highly cited clinical research findings of the past 15 years, 99 percent of molecular research had subsequently been refuted. Epidemiology findings had been contradicted in four-fifths of the cases he looked at, and the usually robust outcomes of clinical trials had a refutation rate of one in four.

The revelations struck a chord with the scientific community at large: A recent essay by Ioannidis simply entitled “Why most published research findings are false” has been downloaded more than 100,000 times; the Boston Globe called it “an instant cult classic.” Now in a Möbius-strip-like twist, there is a growing body of research that is investigating, analyzing, and suggesting causes and solutions for faulty research.

Two papers published this spring in the open-access journal PLoS Medicine by Benjamin Djulbegovic from the University of South Florida and Ramal Moonesinghe from the CDC have delved into the issues raised by Ioannidis and suggested possible ways to mitigate this apparent failure of scientific enterprise. One of the suggestions is to ensure that experimental results are independently replicable. “More often than not, genuine replication is not done, and what we end up with in the literature is corroboration or indirect supporting evidence,” says Moonesinghe.

The culprits appear to be the proverbial suspects: lies, damn lies, and statistics. Jonathan Sterne and George Smith, a statistician and an epidemiologist from the university of Bristol in the UK, point out in a study in British Medical Journal that “the widespread misunderstanding of statistical significance is a fundamental problem” in medical research. What’s more, the scientist’s bias may distort statistics. Pressure to publish can lead to “selective reporting;” the implication is that attention-seeking scientists are exaggerating their results far more often than the occasional, spectacular science fraud would suggest.

Cash-for-science practices between the nutrition and drug companies and the academics that conduct their research may also be playing a role. A survey of published results on beverages earlier this year found that research sponsored by industry is much more likely to report favorable findings than papers with other sources of funding. Although not a direct indication of bias, findings like these feed suspicion that the cherry-picking of data, hindrance of negative results, or adjustment of research is surreptitiously corrupting accuracy. In his essay, Ioannidis wrote, “The greater the financial and other interest and prejudices in a scientific field, the less likely the research findings are to be true.”

Academic bias could also be to blame. As Ioannidis puts it, “Prestigious investigators may suppress via the peer-review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma.” Advocates of prevailing paradigms have been observed to band together in opposition against alternative ideas with perhaps more antagonism than one might expect from objective scientific debate. And the opposition isn’t limited to publication of new science; jobs and grants are also more easily allocated to those affiliated with the scientific party in power.

Ioannidis is adamant that the problem is widespread. “I have heard from scientists from many different fields who think that the problems are the same in their fields as well,” he says. “This is a potentially severe crisis, unless we realize the issue and try to address it.”

With the debate over the causes and solutions of high rates of falsifiable research findings ongoing, how the problem is seen in the eyes of a skeptical public may be another issue altogether. Virginia Barbour, managing editor of PLoS Medicine, puts it simply: “In terms of perception, the point is that science doesn’t emerge from single new findings that become ‘breakthrough’ stories in the media, but rather from developments that mature over months or years, with different sources of experimental validation.”

Originally published May 21, 2007

Tags bias data research

Share this Stumbleupon Reddit Email + More

Now on SEEDMAGAZINE.COM

  • Ideas

    I Tried Almost Everything Else

    John Rinn, snowboarder, skateboarder, and “genomic origamist,” on why we should dumpster-dive in our genomes and the inspiration of a middle-distance runner.

  • Ideas

    Going, Going, Gone

    The second most common element in the universe is increasingly rare on Earth—except, for now, in America.

  • Ideas

    Earth-like Planets Aren’t Rare

    Renowned planetary scientist James Kasting on the odds of finding another Earth-like planet and the power of science fiction.

The Seed Salon

Video: conversations with leading scientists and thinkers on fundamental issues and ideas at the edge of science and culture.

Are We Beyond the Two Cultures?

Video: Seed revisits the questions C.P. Snow raised about science and the humanities 50 years by asking six great thinkers, Where are we now?

Saved by Science

Audio slideshow: Justine Cooper's large-format photographs of the collections behind the walls of the American Museum of Natural History.

The Universe in 2009

In 2009, we are celebrating curiosity and creativity with a dynamic look at the very best ideas that give us reason for optimism.

Revolutionary Minds
The Interpreters

In this installment of Revolutionary Minds, five people who use the new tools of science to educate, illuminate, and engage.

The Seed Design Series

Leading scientists, designers, and architects on ideas like the personal genome, brain visualization, generative architecture, and collective design.

The Seed State of Science

Seed examines the radical changes within science itself by assessing the evolving role of scientists and the shifting dimensions of scientific practice.

A Place for Science

On the trail of the haunts, homes, and posts of knowledge, from the laboratory to the field.

Portfolio

Witness the science. Stunning photographic portfolios from the pages of Seed magazine.

SEEDMAGAZINE.COM by Seed Media Group. ©2005-2015 Seed Media Group LLC. All Rights Reserved.

Sites by Seed Media Group: Seed Media Group | ScienceBlogs | Research Blogging | SEEDMAGAZINE.COM