Voodoo That Scientists Do

Peer Review / by Jon Bardin /

When findings are debated online, as with a yet to be released paper that calls out the field of social neuroscience, who wins?

Page 1 of 2

A debate on the internet over findings released before their formal publication date raises important questions for science.

Few endeavors have been affected more by the tools and evolution of the internet than science publishing. Thousands of journals are available online, and an increasing number of science bloggers are acting as translators, often using lay language to convey complex findings previously read only by fellow experts within a discipline. Now, in the wake of a new paper challenging the methodology of a young field, there is a case study for how the internet is changing the way science itself is conducted.

That area of research is the burgeoning subfield of social neuroscience, which seeks to understand the neurobiological basis of social behavior. Using neuroimaging techniques such as fMRI, researchers correlate neural activity with social and behavioral measures in order to pinpoint areas of the brain associated with social decision making or emotional reactivity.

Late last year, Ed Vul, a graduate student at MIT working with neuroscientist Nancy Kanwisher and UCSD psychologist Hal Pashler, prereleased “Voodoo Correlations in Social Neuroscience” on his website. The journal Perspectives in Psychological Science accepted the paper but will not formally publish it until May.

The paper argues that the way many social neuroimaging researchers are analyzing their data is so deeply flawed that it calls into question much of their methodology. Specifically, Vul and his coauthors claim that many, if not most, social neuroscientists commit a nonindependence error in their research in which the final measure (say, a correlation between behavior and brain activity in a certain region) is not independent of the selection criteria (how the researchers chose which brain region to study), thus allowing noise to inflate their correlation estimates. Further, the researchers found that the methods sections that were clearing peer review boards were woefully inadequate, often lacking basic information about how data was analyzed so that others could evaluate their methods. (Read Vul et al.‘s entire in-press paper here.)

A number of online science writers and bloggers, including the widely read Sharon Begley of Newsweek, immediately wrote about the paper. The vast majority of the online responses to the paper were extremely positive, with Begley suggesting that “like so many researchers in the social sciences, psychologists have physics envy, and think that the illusory precision and solidity of neuroimaging can give their field some rigor.” Vaughan Bell of Mind Hacks predicted that the paper “has the potential to really shake up the world of social cognitive neuroscience.”

In the paper, Vul and his coauthors cite specific studies, many of which were published in leading journals such as Nature and Science, going so far as to call some of the studies “entirely spurious.” Suddenly, a number of researchers found themselves under attack. The paper began filling neuroscientists’ inboxes. Two groups of neuroimaging scientists, shocked by the speed with which this paper was being publicly disseminated, wrote rebuttals and posted them in the comments section of several blogs, including Begley’s. Vul followed up in kind, linking to a rebuttal of the rebuttals in the comment sections of several blogs. This kind of scientific discourse — which typically takes place in the front matter of scholarly journals or over the course of several conferences — developed at a breakneck pace, months before the findings were officially published, and among the usual chaos of blog comments: inane banter, tangents, and valid opinions from the greater public.

Tor Wager, a Columbia University cognitive neuroscientist, whose work was not mentioned in Vul’s paper but who helped prepare one of the rebuttals, says that it was important to respond both publicly and swiftly. “The public and the news media operate on sound bites, and the real scientific issues are quite complex.” His complaints focus not only on the content of Vul’s paper, but also on the authors’ diction — specifically, the title, and its use of “voodoo.”

“When the conversation gets complex — and with statistics it always is — many blog readers will form opinions based on very simple things,” says Wager. “Like words such as ‘voodoo correlations.’ There’s no reason to use such loaded words when making a statistical argument. The argument should be able to stand on its own.”

Page 1 of 2

Tags bias consensus limits medicine neuroscience research

Share this Stumbleupon Reddit Email + More

Now on SEEDMAGAZINE.COM

  • Ideas

    I Tried Almost Everything Else

    John Rinn, snowboarder, skateboarder, and “genomic origamist,” on why we should dumpster-dive in our genomes and the inspiration of a middle-distance runner.

  • Ideas

    Going, Going, Gone

    The second most common element in the universe is increasingly rare on Earth—except, for now, in America.

  • Ideas

    Earth-like Planets Aren’t Rare

    Renowned planetary scientist James Kasting on the odds of finding another Earth-like planet and the power of science fiction.

The Seed Salon

Video: conversations with leading scientists and thinkers on fundamental issues and ideas at the edge of science and culture.

Are We Beyond the Two Cultures?

Video: Seed revisits the questions C.P. Snow raised about science and the humanities 50 years by asking six great thinkers, Where are we now?

Saved by Science

Audio slideshow: Justine Cooper's large-format photographs of the collections behind the walls of the American Museum of Natural History.

The Universe in 2009

In 2009, we are celebrating curiosity and creativity with a dynamic look at the very best ideas that give us reason for optimism.

Revolutionary Minds
The Interpreters

In this installment of Revolutionary Minds, five people who use the new tools of science to educate, illuminate, and engage.

The Seed Design Series

Leading scientists, designers, and architects on ideas like the personal genome, brain visualization, generative architecture, and collective design.

The Seed State of Science

Seed examines the radical changes within science itself by assessing the evolving role of scientists and the shifting dimensions of scientific practice.

A Place for Science

On the trail of the haunts, homes, and posts of knowledge, from the laboratory to the field.

Portfolio

Witness the science. Stunning photographic portfolios from the pages of Seed magazine.

SEEDMAGAZINE.COM by Seed Media Group. ©2005-2015 Seed Media Group LLC. All Rights Reserved.

Sites by Seed Media Group: Seed Media Group | ScienceBlogs | Research Blogging | SEEDMAGAZINE.COM