What happens when research bypasses the validation process and goes straight to the public?

peerreview.jpg Credit: Tray Butler

The whirlwind pace of innovation in the 21st century conceals a dark secret of science: Research is generally at least a year old before it appears in scientific journals.

Publishing a study is often a downright lethargic process: Researchers with a hot new result submit an article to a journal, wait a minimum of three months for experts in the field to review the findings and then spend another few months revising to the reviewers’ recommendations. When a paper is finally accepted, it is edited again and checked by the authors, only to linger—often for several more months—for its turn at the printer.

The process can take so long that some scientists have decided to avoid it altogether.

UCLA brain researcher Marco Iacoboni conducted an “instant-science” study, measuring the brain response of five participants while they watched ads from last month’s Super Bowl. Rather than publishing his results a year or more later, they appeared the next day on the website of the Edge Foundation, a discussion forum for scientists, philosophers, and other prominent intellectuals. Iacoboni admitted that the article wasn’t publishable in a journal, but the point was to show how scientific methods could be used to produce immediate feedback for advertisers.

Given the recent hot news about fraudulent stem cell research in South Korea and cancer research in Norway, it appears that scientists are being pulled in two directions when deciding how to disseminate their results: On the one hand, they want to get the information out as quickly as possible, but on the other hand, they understand that other scientists need to verify that information to ensure the results are sound.

“I don’t mind others seeing my pre-reviewed preprint. I sometimes send an important article to a few colleagues in my field,” said University of Iowa perception researcher Andrew Hollingworth, who allows his peers to take his research into account as they pursue their own work. But Hollingworth said he would be “uncomfortable going to the media without peer review. While casual readers of a newspaper might not understand the peer review process, they would assume research published there has been examined in detail.”

Emory psychologist Drew Westen waited a bit longer than UCLA’s Iacoboni to publicize the results of his study on the 2004 presidential election. He and four other researchers performed brain imaging on a group of volunteers selected for their strong identification with either the Democratic or Republican parties.

Fourteen months after the election, in a January 24th press release on Emory’s web site, Westen announced that when political partisans encountered a statement by their favored candidate that contradicted his public position, they observed little activity in the rational centers of the brain.

“What we saw instead was a network of emotion circuits lighting up,” Westen said in the press release.

Westen’s study hasn’t been published. Emory released the news in anticipation of his presentation of preliminary results at the January 28th meeting of the Society for Personality and Social Psychology. That didn’t stop newspapers, like The New York Times and The Washington Post, from reporting on the study. In addition, the popular blog Slashdot posted the results, attracting over 800 comments. Over 300 other sites featured the findings, with precious few noting that the research hadn’t actually been published. Westen expected the release to gain notice, but says he’s been “shocked” at how much buzz the story has generated.

“We would have loved to have this data out right after the election,” Westen said, noting that instead he submitted it to a respected neuroscience journal. “We had it peer reviewed—went through first round, got positive reviews—and we’re now revising the article.”

In addition, a summary of the article was reviewed before he was allowed to speak at the January conference. “At that point, we felt comfortable with strong data, and decided to make it public,” he said.

That’s a borderline case for Hollingworth, who has declined to release un-reviewed results to the public.

“A researcher has a responsibility to the larger public, to make sure what they’re reporting is valid,” he said. “It’s not always possible for the researcher to know if the work has a fatal flaw, and the peer review process provides an important source of external validation.”

Westen says that there are cases where it may be appropriate to forego waiting for publication: At the same conference where he presented his findings, a respected scholar shared preliminary results that individuals tend to become more politically conservative when reminded of their own mortality.

“When someone’s got a track record of 20 years of impeccable research, and they present a result like this, it would be wrong not to publicize it,” Westen said.

Even when research does pass through formal academic channels, the process isn’t flawless: The two recent highly publicized instances of research fraud in South Korea and Norway were published in peer-reviewed journals.

“Sometimes more specialized journals have a more rigorous peer review process than more general ones,” said Hollingworth, indicating that the more-esteemed journals occasionally try to attract the mainstream media. Hollingworth declined to name specific journals, but The New York Times recently reported that, “among the most prestigious science journals that reporters consult regularly are Nature, Science, The New England Journal of Medicine and The Journal of the American Medical Association.”
The actual reporting in the media can compound the problem. “I have no way to control how journalists frame the article,” says Westen. “Even though researchers may carefully delineate the limitations of their findings, that’s not necessarily going to stop journalists from trumpeting a ‘new breakthrough.’”

Originally published March 12, 2006

Tags

Share this Stumbleupon Reddit Email + More

Now on SEEDMAGAZINE.COM

  • Ideas

    I Tried Almost Everything Else

    John Rinn, snowboarder, skateboarder, and “genomic origamist,” on why we should dumpster-dive in our genomes and the inspiration of a middle-distance runner.

  • Ideas

    Going, Going, Gone

    The second most common element in the universe is increasingly rare on Earth—except, for now, in America.

  • Ideas

    Earth-like Planets Aren’t Rare

    Renowned planetary scientist James Kasting on the odds of finding another Earth-like planet and the power of science fiction.

The Seed Salon

Video: conversations with leading scientists and thinkers on fundamental issues and ideas at the edge of science and culture.

Are We Beyond the Two Cultures?

Video: Seed revisits the questions C.P. Snow raised about science and the humanities 50 years by asking six great thinkers, Where are we now?

Saved by Science

Audio slideshow: Justine Cooper's large-format photographs of the collections behind the walls of the American Museum of Natural History.

The Universe in 2009

In 2009, we are celebrating curiosity and creativity with a dynamic look at the very best ideas that give us reason for optimism.

Revolutionary Minds
The Interpreters

In this installment of Revolutionary Minds, five people who use the new tools of science to educate, illuminate, and engage.

The Seed Design Series

Leading scientists, designers, and architects on ideas like the personal genome, brain visualization, generative architecture, and collective design.

The Seed State of Science

Seed examines the radical changes within science itself by assessing the evolving role of scientists and the shifting dimensions of scientific practice.

A Place for Science

On the trail of the haunts, homes, and posts of knowledge, from the laboratory to the field.

Portfolio

Witness the science. Stunning photographic portfolios from the pages of Seed magazine.

SEEDMAGAZINE.COM by Seed Media Group. ©2005-2015 Seed Media Group LLC. All Rights Reserved.

Sites by Seed Media Group: Seed Media Group | ScienceBlogs | Research Blogging | SEEDMAGAZINE.COM