RECALIBRATION OF SCORING IN DRG STUDY SECTIONS
The primary service that peer review provides to the NIH is advice on the scientific merit of individual
applications so that informed funding decisions can subsequently be made. This advice is provided both
in the form of narrative critique and numerical score. Unfortunately, as scores have become more and
more compressed into a narrow range over time, their discriminatory value has diminished. For example,
in some DRG study sections, a spread of only 5-8 priority score points now covers a range from the 10th
to the 20th percentile. Reviewers' written critiques may also be unduly affected, to the extent that
reviewers feel compelled to adjust their critiques to match the numerical score. Consequently, institute
staff are denied the full benefit of reviewers' scientific judgment in distinguishing among applications.
In order to address this problem, DRG is instituting changes in review procedures, beginning with study
section meetings held this summer, for reviews that will be considered by September and October
meetings of the National Advisory Councils and Boards of the Institutes and Centers for fiscal year 1997
funding.
- Reviewers will be asked to recalibrate their individual ratings such that the "average" application
they customarily review in their study section receives a score of 2.5. Thus, half of all applications
reviewed by a study section should receive scores ranging between 1.0 and 2.5, while the
remaining half are unscored (or in some instances, e.g., review of SBIRs, receive scores
numerically poorer than 2.5).
- Reviewers will be expected to distribute scores, for applications judged to be in the top half, over
the full range from 1.0 to 2.5.
- This change in scoring policy will be implemented in a manner that will neither advantage nor
disadvantage previous applicants. Thus, these changes will begin with the reviews of the first
applications being considered for funding in fiscal year 1997, and will not be influenced by earlier
scoring practices.
- Only applications being reviewed during the June/July 1996 meetings of study sections will be
used to compute percentile ranks for those applications; previous scoring history will be
eliminated. (For applications being reviewed in October/November 1996, percentile rank will be
based on those reviews PLUS those of the previous June/July; and percentiles for applications
reviewed in February/March 1997 will be calculated using that meeting's scores PLUS those from
the previous June/July and October/November meetings.)
Last Update: May 1, 1996
