Page 2 of 4
Rethinking the Metrics of Success
Though the details of science’s infrastructure have changed over the past two millennia, the channels through which individuals and their work are assessed and rewarded have always been about impact, specifically through academic publishing. But for scientists, impact is primarily relevant to their own cloistered fields rather than the broader scientific or global communities.
And while prolific publishing is considered important, the volume of a scientist’s work alone cannot be taken as a measure of relevance to his or her discipline. Instead, judgment lies with the degree of influence that a scientist’s ideas have on the generation of new ones. This process is manifest in the practice of citing earlier work in academic publishing; it is no accident that such citations are the dominant data points in measuring scholarly impact.
The metric at the core of these measurements is the Impact Factor. Conceived by bibliometrician Eugene Garfield in 1972, it is the ratio of citable items in a journal to the number of times they are actually cited over the preceding two years. Impact Factor measures only the importance of a journal as a whole—publication in a high-impact journal is a boon to an individual researcher’s status, but only by proxy.
Other citation-based measures have been suggested to address this problem. One notable example is h-index, which ascribes an influence score to scientists rather than journals or individual papers. While this scheme gets closer to directly rewarding a scientist for the perceived importance of his or her work, the ostensible impact gauged by citation-based measurements remains almost entirely concentrated within a single discipline. Citation practices vary considerably from field to field within academia, meaning metrics that use them are unable to objectively compare, for instance, physicists to molecular biologists.
This is problematic for transferring science beyond the bounds of academia. If we want to know what scientific ideas are influencing decisions and policymaking in the public sphere or in disparate scientific fields, rather than simply the discipline in which an idea originated, citations are of less relevance. The people who must be impacted most urgently by new scientific thinking—civil servants, investors, educators—generally do not publish their work in academic journals. They might as well not exist, as far as the dominant status indicators in the science community are concerned. Writing in the popular press is equally unlikely to garner citations. Even trying to translate research into something more digestible by a lay audience within the academic publishing world is a dead end; editorial and other journalistic material is generally deemed “uncitable.”
Though it is by no means the only aspect of scientific culture responsible, the fixation on citations as a measure of scholarly impact has given scientists few reasons to communicate the value of their work to non-scientists.
“If you’re only interested in what authors of journal articles think about your work, citations will do,” says Johan Bollen, associate professor at Indiana University’s School of Informatics and Computing. “But increasingly, people are looking at metrics that can tell us something more about the broader impact.”
What Bollen and like-minded researchers are looking at is raw usage data for scientists’ work. When the first Impact Factor scores were generated in the late 1970s, a post-hoc analysis of citations made more sense; citations in subsequent journal articles were a concrete record of which scientists had read and approved of each other’s publications. But with the internet turning traditional scientific publishing on its ear, new kinds of records and databases are possible.
While working as a staff scientist at Los Alamos National Laboratory, Bollen began analyzing the scientific works his lab mates were interested in simply by gathering digital records of what they read online. In theory, this method allows near–real time recognition of impact. This idea expanded onto the world stage and became MESUR—Metrics from Usage of Scholarly Resources. By negotiating with publishing companies and other scholarly article aggregators, Bollen was able to collect data on more than a billion usage events for articles and journals between 2002 and 2006.
From this data, Bollen and his colleagues generated network diagrams of how scholarly work is used worldwide. MESUR applied more than 40 different metrics to both usage and citation data across this network, then compared them to see which were best suited to describing different kinds of impact. While the traditional Impact Factor did stand up as a good measure of popularity, the ability to contrast and combine multiple metrics provides for a multidimensional analysis that would be all but impossible without usage data.
According to Bollen, the problem with Impact Factor is that we have a monoculture, one number that everybody seems to chase. “If you have a bunch of different metrics, and they each embody different aspects of scholarly impact, I think that’s a much healthier system. People’s true value can be gleaned, and that should lead to a more diverse, more interdisciplinary scientific landscape.”
Another advantage of usage-based metrics is that they extend beyond the walls of academia. Current usage records don’t fully capture who is doing the using, but the technology to do so certainly exists. Metrics could some day pinpoint where and when a scientist’s work is being considered or adopted in public contexts.
Equally important is the ability for non-academic publishing to be similarly analyzed. Instead of being viewed purely as a liability or distraction, a scientist’s attempts to connect his or her research with the public could be measured for impact and thus become more quantitatively integrated into tenure and funding decisions.
After a two-year hiatus, MESUR is collecting data again and will forge ahead on this path. “Non-traditional publications, whether they are newspapers or popular science magazines, can be incorporated into this map of science; we can get usage data on them, too,” says Bollen. “Then we could see what their actual role is in the fabric of science, because to claim that they have none is ludicrous.”
Developing standards for deeper and universally applicable metadata on usage goes hand in hand with the movement toward transparency and interoperability in online publishing. Open access publishing, which makes journal articles freely available to all via the web, epitomizes these values. The Public Library of Science, the premier open access journal, has recently announced that it will make usage data on its articles freely available as well. And more traditional publishers are now collaborating on the Open Researcher and Contributor ID, a standard identifier for individuals meant to help track the value of their work directly, rather than by the proxies of group articles or whole journals.
As these movements and tools grow—and a new philosophy behind measuring the reach and impact of scientific work takes shape—a clearer picture will emerge of how science flows through the world. And it is only with such a picture that this flow can be better directed.
Page 2 of 4