Credit: Flickr user Stefano Mazzone
Women married to wealthier men say they experience more orgasms. People who say they watch more TV are more likely to die sooner. People who say they got more vaccines are more likely to say they got sick after getting them. People who say they eat more fish are more likely to say they are depressed.
All these statements are true, backed up by scientific research. But the studies that produced these surprising results all share the same weakness: They rely on self-reporting. Think of it this way: Just because you say you always have great sex doesn’t make it true.
Psychology researchers have debated the value of self-reporting for as long as the field of psychology has existed. In 1913, in what became known as the Behaviorist Manifesto, John Watson decried one form of self-reporting, introspection. Watson believed that in order for psychology to be a true science, it should not use self-reports at all. Instead, scientists should directly study behavior. Just as we study a rat’s visual system by observing what type of light it responds to, Watson argued that we should study human psychology solely through systematic behavioral experiments.
Watson was responding to research methods that were in vogue in the early 20th century. Freud and others were gaining tremendous notoriety making astonishing claims about human “consciousness” based on scant evidence. Freud couldn’t offer any evidence that humans possessed an id, ego, and superego; he simply asserted it based on what his patients reported to him and his own introspection. Watson felt that if psychology continued to move in this direction, it could never be taken seriously as a science.
But aren’t there some occasions when self-reported data is better than nothing at all? After all, it’s a lot easier to ask someone how much TV they watch than it is to look over their shoulder for a week and find out exactly. If people who say they watch a lot of TV really are more likely to die than those who don’t, isn’t that important to know—even if their recollections aren’t perfect?
Not when that research results in misleading conclusions, argues the cancer surgeon who blogs as “Orac.” The popular anti-vaccination movement arose primarily on the basis of a few specious self-report studies purportedly linking vaccinations to poor health. In 2008, Orac reported on a study that compared self-reports to real medical records. British troops deploying for Iraq from 2004-2006 were routinely administered a number of vaccines. When they were asked about those vaccinations later, the individuals who said they received more vaccinations in a day also reported having more health problems. But when the researchers examined the medical records themselves, they found no relationship between the number of vaccinations given and the reported health problems. So while a belief that more vaccinations were given was associated with health problems, actually receiving more vaccinations was not. The research was published in the British Medical Journal in 2008. But remember, the health problems were self-reported too, so there may not even be a relationship between believing you receive vaccines and having real health problems.
So is there any situation in which self-reporting can offer useful information for researchers? Graduate students Jason Goldman and Travis Saunders discussed this question on the Obesity Panacea podcast last week. Saunders studies obesity and physical fitness, while Goldman does developmental psychology research. Saunders is interested in how much time people spend in exercise versus sedentary activity, and is skeptical of the role of self-reports. Goldman, on the other hand, says that self-reporting can be the “bread and butter” of some areas of psychology. It’s still really the only way to get a sense of how a person thinks and feels, Goldman says, and it can be a useful tool, as long as the researcher is aware of the dangers and makes an effort to account for them. Saunders agrees, and points to important research showing that students who are in classes with more overweight and obese individuals are less likely to believe themselves to be overweight. Clearly, if students don’t see themselves as overweight, it’s going to affect how teachers and health professionals address the problem.
But the problems with self-reports go beyond the imperfect perceptions and memories of respondents. Goldman and Saunders point to a 1999 report by Norbert Schwarz, who says that even the phrasing of questions in surveys can have a dramatic affect on the results. Goldman notes that, “If I asked you in an open-ended question what you did today, you probably wouldn’t say you took a shower. But if I gave you a checklist of items, one of which was ‘took a shower,’ you would certainly check it off.” But Saunders says it also makes you very unlikely to mention something that’s not in the list of options. “Even if you give people a 10-point scale, if you offer a 0-10 rating, you’ll get different answers from a scale that ranges from -5 to +5.” There are cultural problems, language issues, and other variables that make it extremely difficult to compare data from self-report studies. Even so, Schwarz remains optimistic about the use of self-reporting. Because we know so much about the potential problems, we can use this knowledge to create better surveys and calibrate the results.
Ultimately, some of the best ways to learn about real-world behavior of individuals may come not from direct self-report, but from indirect measures. While asking someone how many books they read last year might not be a very reliable measure of reading experience, Goldman says, we can measure that ability indirectly by giving them a quiz about books and their authors. In the aggregate, people who do better on the quiz are probably better readers, and we can start to learn why those people are better at reading—and perhaps how to make our schools better too.
These indirect measures are actually closer to Watson’s ideal of psychology as a study of behavior than Freud’s introspection. They are repeatable and controllable, which are the hallmarks of a true scientific endeavor. But that doesn’t mean psychologists and other researchers shouldn’t try to learn how people believe they’re thinking—that knowledge, in and of itself, can be scientifically useful. Rather, scientists simply need to know the limitations of self-reporting as they conduct their research.
Dave Munger is editor of ResearchBlogging.org, where you can find thousands of blog posts on this and myriad other topics. Each week, he writes about recent posts on peer-reviewed research from across the blogosphere. See previous Research Blogging columns »
Originally published June 23, 2010
