Peer Pressure

Europe / by James Wilsdon /

James Wilsdon takes a close look at Britain's system for defining excellence at the country's top-performing universities.

With $2.2 billion up for grabs based on its findings, it’s hardly surprising that the 2008 Research Assessment Exercise (RAE) has dominated conversations in Britain’s universities in recent months. Undertaken once every seven years, this audit of Britain’s research base judged the quality of the work submitted by more than 52,000 academics from 159 universities. Making sense of what happens this month, when their scores are translated into hard cash, has become a science of its own.

Policymakers and scientists around the world are waiting to see how this assessment plays out, as a number of places, such as Australia and New Zealand, have research funding systems modelled directly on the UK’s. But the underlying dilemmas are the same everywhere: How do you measure, compare, and reward excellence in research? For that matter, how do you define excellence? And how do you balance the freedoms vital for groundbreaking research with the need for financial accountability?

The British RAE has relied primarily on peer review to answer those questions, with subject-based panels of scientists and other academics ranking the submissions of their colleagues. But the system is set to change with the 2015 exercise, focusing more on hard data and quantitative outputs. Whether these changes will help or hinder the production of excellent research is now the subject of strenuous debate across Europe.

The good news is that Britain’s research base currently appears to be in robust health. David Eastwood, chief executive of the Higher Education Funding Council for England, which coordinated the RAE, described the results as representing “an outstanding achievement.” With more than half the applicants’ work assessed as either “world leading” or “internationally excellent,” the exercise confirms that “the UK is among the top rank of the research powers in the world.”

Oxford. Credit: Andreas Stirnberg/Getty Images

However, all of this success has complicated matters. In previous rounds of research assessment, the premier league of 25 or so research-intensive universities has scooped up 80 percent of the available funding. This time around, with “worldleading” researchers identified in almost 50 institutions, there are fears that money will be spread around too thinly, weakening established centers of scientific excellence. Some have complained of grade inflation, arguing that the peer-review process has resulted in the highest scores being given too generously.

Those kinds of complaints have led to the changes slated for the 2015 RAE, which will for the first time examine factors like citation rates and the economic impacts of the research in question. This move has been controversial, with year-spanning debates raging over how different models might benefit or harm particular disciplines, and the perverse incentives that could result.

Some see the reliance on peer review within the funding system as a bigger part of the problem. In a joint letter to the Financial Times last year, a group of 25 senior British and American scientists, including two Nobel Laureates, described it as “a policy that works well enough for the mainstream but fails at the margins where unpredictable and transformative discoveries are made.” The group called for the creation of a new “Planck Club” of scientists, who would receive the funding to pursue breakthroughs, unconstrained by the strictures of peer review.

But if a model based on established scientists’ opinions has its limitations, one based on citations could be even more problematic. Citation rates vary widely across disciplines and over time, with some papers sparking a short burst of interest, and others having a slower yet deeper impact on their field. A recent editorial in Nature noted that its most highly cited paper in 2007 was a pilot study in screening for functional elements of the human genome. This was of interest to others because of the technique being used, but would hardly qualify as groundbreaking science.

In an ideal world, perhaps, great scientists would simply get a blank check and a green light. The problem remains, of course, selecting which scientists deserve access to limited resources. Back on Earth, metrics may have a role to play in the selection process, and the 2015 RAE could illuminate what quantitative measures work and which should be forgotten.

It’s likely, however, that some form of peer review will always be important in bringing judgment, experience, and insight to bear on new work. The challenge for policymakers is in designing a system that rewards excellence, nurtures promise, and gives room for creativity to flourish.

Originally published March 13, 2009

Tags decision making education funding policy politics

Share this Stumbleupon Reddit Email + More

Now on SEEDMAGAZINE.COM

  • Ideas

    I Tried Almost Everything Else

    John Rinn, snowboarder, skateboarder, and “genomic origamist,” on why we should dumpster-dive in our genomes and the inspiration of a middle-distance runner.

  • Ideas

    Going, Going, Gone

    The second most common element in the universe is increasingly rare on Earth—except, for now, in America.

  • Ideas

    Earth-like Planets Aren’t Rare

    Renowned planetary scientist James Kasting on the odds of finding another Earth-like planet and the power of science fiction.

The Seed Salon

Video: conversations with leading scientists and thinkers on fundamental issues and ideas at the edge of science and culture.

Are We Beyond the Two Cultures?

Video: Seed revisits the questions C.P. Snow raised about science and the humanities 50 years by asking six great thinkers, Where are we now?

Saved by Science

Audio slideshow: Justine Cooper's large-format photographs of the collections behind the walls of the American Museum of Natural History.

The Universe in 2009

In 2009, we are celebrating curiosity and creativity with a dynamic look at the very best ideas that give us reason for optimism.

Revolutionary Minds
The Interpreters

In this installment of Revolutionary Minds, five people who use the new tools of science to educate, illuminate, and engage.

The Seed Design Series

Leading scientists, designers, and architects on ideas like the personal genome, brain visualization, generative architecture, and collective design.

The Seed State of Science

Seed examines the radical changes within science itself by assessing the evolving role of scientists and the shifting dimensions of scientific practice.

A Place for Science

On the trail of the haunts, homes, and posts of knowledge, from the laboratory to the field.

Portfolio

Witness the science. Stunning photographic portfolios from the pages of Seed magazine.

SEEDMAGAZINE.COM by Seed Media Group. ©2005-2015 Seed Media Group LLC. All Rights Reserved.

Sites by Seed Media Group: Seed Media Group | ScienceBlogs | Research Blogging | SEEDMAGAZINE.COM