How many reviewers should be used?
In PLoS One there is a somewhat interesting article on how many reviewers should be used to review a grant - in this context, it's a NIH grant.
Sample Size and Precision in NIH Peer Review by Kaplan et al.
I am reminded of a debate going on in the IT-sector, whether five testers are enough to ensure that the quality of a website is good enough. The idea was first proposed by Robert Virzi in 1992, and was spread by Jakob Nielsen (see e.g. Why You Only Need to Test With 5 Users), and has been widely criticized by others (e.g. Laura Faulkner in Beyond the five-user assumption: Benefits of increased sample sizes in usability testing (.pdf))
The five tester idea is based on two premises:
1) There are only few resources available (time, money, personnel) , and they should be used most efficiently.
2) Most problems (80% or more) can be found by five testers.
These premises can to some degree probably be rephrased to cover the premises for why using four peer reviewers in the NIH peer review process.
As the IT sector has found out, these premises are not valid, which means that the five tester principles results in bad, or even fatally flawed, products going to the market. Kaplan et al. demonstrates that the premises for using four reviewers are also flawed, and can result in wrong prioritizing of NIH funds.
I cannot claim to have any great insight into the problem, but I hope that the NIH takes this article to heart, and evaluates if there is a better way to ensure proper peer review giving the resource constraints. Kaplan et al suggests using shorter proposals, since that will mean that each reviewer can evaluate a larger number of proposals without spending more time on it. I don't know if this is the right solution, since it might make it harder to actually evaluate the merits of the proposal, but it might be worth looking into.
Sample Size and Precision in NIH Peer Review by Kaplan et al.
I am reminded of a debate going on in the IT-sector, whether five testers are enough to ensure that the quality of a website is good enough. The idea was first proposed by Robert Virzi in 1992, and was spread by Jakob Nielsen (see e.g. Why You Only Need to Test With 5 Users), and has been widely criticized by others (e.g. Laura Faulkner in Beyond the five-user assumption: Benefits of increased sample sizes in usability testing (.pdf))
The five tester idea is based on two premises:
1) There are only few resources available (time, money, personnel) , and they should be used most efficiently.
2) Most problems (80% or more) can be found by five testers.
These premises can to some degree probably be rephrased to cover the premises for why using four peer reviewers in the NIH peer review process.
As the IT sector has found out, these premises are not valid, which means that the five tester principles results in bad, or even fatally flawed, products going to the market. Kaplan et al. demonstrates that the premises for using four reviewers are also flawed, and can result in wrong prioritizing of NIH funds.
I cannot claim to have any great insight into the problem, but I hope that the NIH takes this article to heart, and evaluates if there is a better way to ensure proper peer review giving the resource constraints. Kaplan et al suggests using shorter proposals, since that will mean that each reviewer can evaluate a larger number of proposals without spending more time on it. I don't know if this is the right solution, since it might make it harder to actually evaluate the merits of the proposal, but it might be worth looking into.
Labels: NIH, peer review, PLoS One, science
posted by Kristjan Wager at 3:00 PM
View blog reactions
6 Comments:
Your Kaplan link isn't linking. Off to find the paper on my own...
Okay, so we need to make all NIH proposals pretty much public, and really short, and let everybody vote on them. Okay, not everybody, since many people wouldn't care, but what if everyone working on NIH-funded projects got a vote? I wonder how that would work?
Thanks for the heads up about the link - I had used the trackback link by accident.
Regarding your idea of NIH funded people voting on it - can you say conflict of interest? As a minimum the proposals should be anonymized.
Sure, they can be anonimous. But the current system uses a smallish board of reviewers, top people in their fields, responsible for millions in grants from the NIH to their home university departments. To some extent, anonymity won't work wither, because the top people in mouse microscopy (for example) know the other dozen or so top people in the field.
I'm thinking that instead of just a group of hand-picked, funded reasearchers who are also divion or department chairs, and who are also the same reviews of manuscripts (albeit, anonymously again), maybe it would be better to open it up to a much bigger field, including assistant professors and adjucts and graduate students and staff librarians.
I don't know, I'm just thinking. How do y'all divy up grant funding?
I actually don't know how grant funding works in Denmark. A large part of it goes through the universities, but otherwise, I don't know.
I'll try to follow up on that, and see if I can find out how it happens in Denmark.
Thanks, I'd be interested to know.
Post a Comment
<< Home