How many reviewers should be used?

In PLoS One there is a somewhat interesting article on how many reviewers should be used to review a grant - in this context, it's a NIH grant.

Sample Size and Precision in NIH Peer Review by Kaplan et al.

I am reminded of a debate going on in the IT-sector, whether five testers are enough to ensure that the quality of a website is good enough. The idea was first proposed by Robert Virzi in 1992, and was spread by Jakob Nielsen (see e.g. Why You Only Need to Test With 5 Users), and has been widely criticized by others (e.g. Laura Faulkner in Beyond the five-user assumption: Benefits of increased sample sizes in usability testing (.pdf))

The five tester idea is based on two premises:
1) There are only few resources available (time, money, personnel) , and they should be used most efficiently.
2) Most problems (80% or more) can be found by five testers.

These premises can to some degree probably be rephrased to cover the premises for why using four peer reviewers in the NIH peer review process.

As the IT sector has found out, these premises are not valid, which means that the five tester principles results in bad, or even fatally flawed, products going to the market. Kaplan et al. demonstrates that the premises for using four reviewers are also flawed, and can result in wrong prioritizing of NIH funds.

I cannot claim to have any great insight into the problem, but I hope that the NIH takes this article to heart, and evaluates if there is a better way to ensure proper peer review giving the resource constraints. Kaplan et al suggests using shorter proposals, since that will mean that each reviewer can evaluate a larger number of proposals without spending more time on it. I don't know if this is the right solution, since it might make it harder to actually evaluate the merits of the proposal, but it might be worth looking into.

The future of peer review

The European Science Foundation held a conference about peer review last year, and debated how to make the process better. This was not something I was aware of, until I saw this ScienceDaily article - Quality On Peer Review Must Be Raised With Co-operation, Says Report

Scientists are questioning whether peer review, the internationally accepted form of scientific critique, is able to meet the challenges posed by the rapid changes in the research landscape. A report published by the European Science Foundation (ESF) has showcased a number of options that could lead to a greater openness to innovative research.

The report “Peer review: its present and future states”, which draws on ideas from an international conference in Prague in October 2006, reflects some concern on the shortcomings of peer review while outlines some possible measures to cope with them.

[...]

A central theme of the report is that the current peer review system might not adequately assess the most pioneering research proposals, as they may be viewed as too risky. John O’Reilly, former Chief Executive of the U.K.’s engineering and Physical Sciences Research Council (EPSRC), now Vice Chancellor of Cranfield University, said traditional peer review might be too risk averse. He suggested the need to encourage pioneering research that is high risk in the proposal, but high impact if successful.

The conference called for new approaches, enabling the assessment of innovative research, to be embedded in the peer review system. An example of a new approach to overcome the perceived risk-averse funding culture was given by Dr. He Minghong from the National Natural Science Foundation China. His Foundation encourages reviewers and programme managers to spot risky project proposals which are then funded under stricter conditions. Their duration is shorter, their budget smaller and they are more closely monitored.

I am on two mind on this. There is no doubt that they are right that pioneering research has a hard time in traditional peer reviewed journals, but I can't help worry about the number of bad papers that will be published if there is a less critical review of such papers than there is now. The Chinese approach does sound good, but it can only be applied to studies funded by public grants.

Hopefully the focus on the issue can bring new ideas to the table, which will allow the peer review system to continue, while allowing more room for pioneering works. Peer reviews are critical for proper science.

ScienceDaily unfurtunately don't link to the papers (or even press releases) that they are covering, so I haven't been able to locate the report, “Peer review: its present and future states”. The conference in Prague have a website though, which has the presentations from the conference.

Edited to add: I found both the press release, and a link to the report (.pdf).