James Wilsdon is the director of the Royal Society’s International Science Policy Center. He writes the EUROPE column on current ideas and issues at the forefront of European science.
With $2.2 billion up for grabs based on its findings, it’s hardly surprising that the 2008 Research Assessment Exercise (RAE) has dominated conversations in Britain’s universities in recent months. Undertaken once every seven years, this audit of Britain’s research base judged the quality of the work submitted by more than 52,000 academics from 159 universities. Making sense of what happens this month, when their scores are translated into hard cash, has become a science of its own.
Policymakers and scientists around the world are waiting to see how this assessment plays out, as a number of places, such as Australia and New Zealand, have research funding systems modelled directly on the UK’s. But the underlying dilemmas are the same everywhere: How do you measure, compare, and reward excellence in research? For that matter, how do you define excellence? And how do you balance the freedoms vital for groundbreaking research with the need for financial accountability?
The British RAE has relied primarily on peer review to answer those questions, with subject-based panels of scientists and other academics ranking the submissions of their colleagues. But the system is set to change with the 2015 exercise, focusing more on hard data and quantitative outputs. Whether these changes will help or hinder the production of excellent research is now the subject of strenuous debate across Europe.
The good news is that Britain’s research base currently appears to be in robust health. David Eastwood, chief executive of the Higher Education Funding Council for England, which coordinated the RAE, described the results as representing “an outstanding achievement.” With more than half the applicants’ work assessed as either “world leading” or “internationally excellent,” the exercise confirms that “the UK is among the top rank of the research powers in the world.”
Oxford. Credit: Andreas Stirnberg/Getty Images
However, all of this success has complicated matters. In previous rounds of research assessment, the premier league of 25 or so research-intensive universities has scooped up 80 percent of the available funding. This time around, with “worldleading” researchers identified in almost 50 institutions, there are fears that money will be spread around too thinly, weakening established centers of scientific excellence. Some have complained of grade inflation, arguing that the peer-review process has resulted in the highest scores being given too generously.
Those kinds of complaints have led to the changes slated for the 2015 RAE, which will for the first time examine factors like citation rates and the economic impacts of the research in question. This move has been controversial, with year-spanning debates raging over how different models might benefit or harm particular disciplines, and the perverse incentives that could result.
Some see the reliance on peer review within the funding system as a bigger part of the problem. In a joint letter to the Financial Times last year, a group of 25 senior British and American scientists, including two Nobel Laureates, described it as “a policy that works well enough for the mainstream but fails at the margins where unpredictable and transformative discoveries are made.” The group called for the creation of a new “Planck Club” of scientists, who would receive the funding to pursue breakthroughs, unconstrained by the strictures of peer review.
But if a model based on established scientists’ opinions has its limitations, one based on citations could be even more problematic. Citation rates vary widely across disciplines and over time, with some papers sparking a short burst of interest, and others having a slower yet deeper impact on their field. A recent editorial in Nature noted that its most highly cited paper in 2007 was a pilot study in screening for functional elements of the human genome. This was of interest to others because of the technique being used, but would hardly qualify as groundbreaking science.
In an ideal world, perhaps, great scientists would simply get a blank check and a green light. The problem remains, of course, selecting which scientists deserve access to limited resources. Back on Earth, metrics may have a role to play in the selection process, and the 2015 RAE could illuminate what quantitative measures work and which should be forgotten.
It’s likely, however, that some form of peer review will always be important in bringing judgment, experience, and insight to bear on new work. The challenge for policymakers is in designing a system that rewards excellence, nurtures promise, and gives room for creativity to flourish.
Originally published March 13, 2009
