Questionable Answers

Research Blogging / by Dave Munger /

Many dramatic research results rely on the reports of survey respondents, who are prone to error, exaggeration, and misperception. Is there any place for self-reported results in science?

Credit: Flickr user Stefano Mazzone

Women married to wealthier men say they experience more orgasms. People who say they watch more TV are more likely to die sooner. People who say they got more vaccines are more likely to say they got sick after getting them. People who say they eat more fish are more likely to say they are depressed.

All these statements are true, backed up by scientific research. But the studies that produced these surprising results all share the same weakness: They rely on self-reporting. Think of it this way: Just because you say you always have great sex doesn’t make it true.

Psychology researchers have debated the value of self-reporting for as long as the field of psychology has existed. In 1913, in what became known as the Behaviorist Manifesto, John Watson decried one form of self-reporting, introspection. Watson believed that in order for psychology to be a true science, it should not use self-reports at all. Instead, scientists should directly study behavior. Just as we study a rat’s visual system by observing what type of light it responds to, Watson argued that we should study human psychology solely through systematic behavioral experiments.

Watson was responding to research methods that were in vogue in the early 20th century. Freud and others were gaining tremendous notoriety making astonishing claims about human “consciousness” based on scant evidence. Freud couldn’t offer any evidence that humans possessed an id, ego, and superego; he simply asserted it based on what his patients reported to him and his own introspection. Watson felt that if psychology continued to move in this direction, it could never be taken seriously as a science.

But aren’t there some occasions when self-reported data is better than nothing at all? After all, it’s a lot easier to ask someone how much TV they watch than it is to look over their shoulder for a week and find out exactly. If people who say they watch a lot of TV really are more likely to die than those who don’t, isn’t that important to know—even if their recollections aren’t perfect?

Not when that research results in misleading conclusions, argues the cancer surgeon who blogs as “Orac.” The popular anti-vaccination movement arose primarily on the basis of a few specious self-report studies purportedly linking vaccinations to poor health. In 2008, Orac reported on a study that compared self-reports to real medical records. British troops deploying for Iraq from 2004-2006 were routinely administered a number of vaccines. When they were asked about those vaccinations later, the individuals who said they received more vaccinations in a day also reported having more health problems. But when the researchers examined the medical records themselves, they found no relationship between the number of vaccinations given and the reported health problems. So while a belief that more vaccinations were given was associated with health problems, actually receiving more vaccinations was not.  The research was published in the British Medical Journal in 2008. But remember, the health problems were self-reported too, so there may not even be a relationship between believing you receive vaccines and having real health problems.

So is there any situation in which self-reporting can offer useful information for researchers? Graduate students Jason Goldman and Travis Saunders discussed this question on the Obesity Panacea podcast last week. Saunders studies obesity and physical fitness, while Goldman does developmental psychology research. Saunders is interested in how much time people spend in exercise versus sedentary activity, and is skeptical of the role of self-reports. Goldman, on the other hand, says that self-reporting can be the “bread and butter” of some areas of psychology. It’s still really the only way to get a sense of how a person thinks and feels, Goldman says, and it can be a useful tool, as long as the researcher is aware of the dangers and makes an effort to account for them. Saunders agrees, and points to important research showing that students who are in classes with more overweight and obese individuals are less likely to believe themselves to be overweight. Clearly, if students don’t see themselves as overweight, it’s going to affect how teachers and health professionals address the problem.

But the problems with self-reports go beyond the imperfect perceptions and memories of respondents. Goldman and Saunders point to a 1999 report by Norbert Schwarz, who says that even the phrasing of questions in surveys can have a dramatic affect on the results. Goldman notes that, “If I asked you in an open-ended question what you did today, you probably wouldn’t say you took a shower. But if I gave you a checklist of items, one of which was ‘took a shower,’ you would certainly check it off.” But Saunders says it also makes you very unlikely to mention something that’s not in the list of options. “Even if you give people a 10-point scale, if you offer a 0-10 rating, you’ll get different answers from a scale that ranges from -5 to +5.” There are cultural problems, language issues, and other variables that make it extremely difficult to compare data from self-report studies. Even so, Schwarz remains optimistic about the use of self-reporting. Because we know so much about the potential problems, we can use this knowledge to create better surveys and calibrate the results.

Ultimately, some of the best ways to learn about real-world behavior of individuals may come not from direct self-report, but from indirect measures. While asking someone how many books they read last year might not be a very reliable measure of reading experience, Goldman says, we can measure that ability indirectly by giving them a quiz about books and their authors. In the aggregate, people who do better on the quiz are probably better readers, and we can start to learn why those people are better at reading—and perhaps how to make our schools better too.

These indirect measures are actually closer to Watson’s ideal of psychology as a study of behavior than Freud’s introspection. They are repeatable and controllable, which are the hallmarks of a true scientific endeavor. But that doesn’t mean psychologists and other researchers shouldn’t try to learn how people believe they’re thinking—that knowledge, in and of itself, can be scientifically useful. Rather, scientists simply need to know the limitations of self-reporting as they conduct their research.

Dave Munger is editor of ResearchBlogging.org, where you can find thousands of blog posts on this and myriad other topics. Each week, he writes about recent posts on peer-reviewed research from across the blogosphere. See previous Research Blogging columns »

 

Originally published June 23, 2010

Tags

Share this Stumbleupon Reddit Email + More

Now on SEEDMAGAZINE.COM

  • Ideas

    I Tried Almost Everything Else

    John Rinn, snowboarder, skateboarder, and “genomic origamist,” on why we should dumpster-dive in our genomes and the inspiration of a middle-distance runner.

  • Ideas

    Going, Going, Gone

    The second most common element in the universe is increasingly rare on Earth—except, for now, in America.

  • Ideas

    Earth-like Planets Aren’t Rare

    Renowned planetary scientist James Kasting on the odds of finding another Earth-like planet and the power of science fiction.

The Seed Salon

Video: conversations with leading scientists and thinkers on fundamental issues and ideas at the edge of science and culture.

Are We Beyond the Two Cultures?

Video: Seed revisits the questions C.P. Snow raised about science and the humanities 50 years by asking six great thinkers, Where are we now?

Saved by Science

Audio slideshow: Justine Cooper's large-format photographs of the collections behind the walls of the American Museum of Natural History.

The Universe in 2009

In 2009, we are celebrating curiosity and creativity with a dynamic look at the very best ideas that give us reason for optimism.

Revolutionary Minds
The Interpreters

In this installment of Revolutionary Minds, five people who use the new tools of science to educate, illuminate, and engage.

The Seed Design Series

Leading scientists, designers, and architects on ideas like the personal genome, brain visualization, generative architecture, and collective design.

The Seed State of Science

Seed examines the radical changes within science itself by assessing the evolving role of scientists and the shifting dimensions of scientific practice.

A Place for Science

On the trail of the haunts, homes, and posts of knowledge, from the laboratory to the field.

Portfolio

Witness the science. Stunning photographic portfolios from the pages of Seed magazine.

SEEDMAGAZINE.COM by Seed Media Group. ©2005-2012 Seed Media Group LLC. All Rights Reserved.

Sites by Seed Media Group: Seed Media Group | ScienceBlogs | Research Blogging | SEEDMAGAZINE.COM