Skip to Content


Center for Scientific Review Announcement of Requirements and Registration for New Methods To Detect Bias in Peer Review

Document Details

Information about this document as published in the Federal Register.

Document Statistics
Document page views are updated periodically throughout the day and are cumulative counts for this document including its time on Public Inspection. Counts are subject to sampling, reprocessing and revision (up or down) throughout the day.
Published Document

This document has been published in the Federal Register. Use the PDF linked in the document sidebar for the official electronic format.

Start Preamble Start Authority

Authority: 15 U.S.C. 3719.

End Authority


The Center for Scientific Review (CSR) is seeking ideas for the detection of bias in NIH Peer Review of grant applications in a challenge titled “New Methods to Detect Bias in Peer Review.” This notice provides information regarding requirements and registration for this challenge.


Submission Period: May 5, 2014 through 11:59 p.m. Eastern Time, June 30, 2014.

Judging Period: July 16, 2014 through August 29, 2014.

Winners Announced: September 2, 2014.


Details on the NIH/CSR Peer Review process can be found on the Reviewer Resources tab at For questions about this challenge, email Start Printed Page 25604 or call at 301-300-3839.

Start Further Info


Monica Basco, Center for Scientific Review, phone: 301-300-3839 or email:

End Further Info End Preamble Start Supplemental Information


The mission of CSR is to ensure that the National Institutes of Health (NIH) grant applications receive fair, independent, expert, and timely reviews so NIH can fund the most promising research. For this challenge, “New Methods to Detect Bias in Peer Review,” CSR is seeking ideas for the detection of bias in NIH Peer Review of grant applications. CSR is particularly interested in approaches, strategies, methodologies, and/or measures that would be sensitive to detecting bias among reviewers based on gender, race/ethnicity, institutional affiliation, area of science, and amount of research experience of applicants.

This challenge is consistent with peer review authority under sections 492 and 492A of the Public Health Service Act and federal regulations governing “Scientific Peer Review of Research Grant Applications and Research and Development Contract Projects” (42 CFR Part 52h). The challenge is part of a larger quality assessment activity related to peer review. Research findings (Ginther et al., 2011; 2012) showed a discrepancy in success rates for NIH R01 grant funding between White applicants and Black applicants, raising the question of possible bias in the peer review process. This challenge aims to address that discrepancy by soliciting ideas for detecting potential bias in peer review. It directly supports the mission of CSR to ensure that the best and brightest minds have an equal opportunity to contribute to the realization of our national research goals.

Subject of Challenge: The subject of this challenge is to seek ideas for the detection of bias in NIH Peer Review. The mission of the NIH is to seek fundamental knowledge about the nature and behavior of living systems and to apply that knowledge to enhance health, lengthen life, and reduce the burdens of illness and disability. NIH has a longstanding and time tested system of peer review to identify the most promising biomedical research to accomplish these aims. As the portal for NIH grant applications and their review for scientific and technical merit, CSR is engaged in a new initiative to closely examine the peer review process. Aims include the identification of procedures and practices that are most beneficial in accomplishing CSR's mission as well as identifying any aspects that might make the review process vulnerable to bias. The goal is to enhance the quality and validity of the peer review process.

As background, every grant application submitted to NIH must undergo two levels of NIH Peer Review prior to funding. The first level of review is carried out by a Scientific Review Group (SRG) composed primarily of non-federal scientists who have expertise in relevant scientific disciplines and current research areas. The second level of review is performed by Institute and Center National Advisory Councils or Boards who make recommendations on priority areas of research, pending policy, and funding of particular applications. Councils are composed of both scientific and public representatives chosen for their expertise, interest, or activity in matters related to health and disease. Only applications that are recommended for approval by both the SRG and the Advisory Council may be recommended for funding. Final funding decisions are made by the director of the relevant NIH Institute or Center.

CSR strives to ensure that NIH grant applications receive fair, independent, expert, and timely reviews—free from inappropriate influences—so NIH can fund the most promising research. However, recent studies (Ginther et al., 2011; 2012) have shown that African American researchers are less likely than White researchers to receive NIH R01 grant funding by at least 10 percentage points. An investigation of racial disparities in grant funding must include the exploration of potential bias in the peer review system.

There are several challenges in the assessment of bias in peer review. Any reactive effects of assessing racial bias must be minimized. That is; detection strategies should not have a detrimental effect on reviewers by creating a sensitivity that did not previously exist. And, while the written critiques of reviewers may provide opportunities to identify biased comments, because of the confidential nature of peer review, the names and demographic characteristics of reviewers assigned to specific applications are not retained and not all grant applicants and reviewers provide sensitive demographic data such as race and ethnicity.

In this challenge, participants are asked to submit their ideas for the detection of possible bias in the NIH Peer Review process. These ideas should provide approaches, strategies, methodologies, and/or measures that would be sensitive to detecting bias among reviewers due to gender, race/ethnicity, institutional affiliation, area of science, and prior funding of grant applicants (See complete submission requirements below).

Eligibility Rules for Participating in the Challenge: The challenge is open to any individual, group of individuals, or entity (each referred to in this notice as a participant) who meets the eligibility criteria below. There is no limit to the number of entries a participant can submit.

To be eligible to win a prize under this challenge:

(1) The participant shall have registered to participate in the competition under the rules promulgated by CSR as described in this notice.

(2) The participant (including each individual participating as a member of a group participant) shall have complied with all the requirements under this section.

(3) In the case of a private entity, the entity shall be incorporated in and maintain a primary place of business in the United States, and in the case of an individual, whether participating singly or in a group, each shall be a citizen or permanent resident of the United States.

(4) Individuals (whether competing alone or part of a group) who are younger than 18 must have their parent or legal guardian complete the Parental Consent Form. The form can be found on the Challenge Web page at

(5) The participant may not be a Federal entity or Federal employee acting within the scope of his or her employment.

(6) The participant shall not be an HHS employee working on their applications or submissions during assigned duty hours.

(7) The participant shall not be an employee of the National Institutes of Health, the Center for Scientific Review, a member of the Subcommittee on Peer Review or any other party involved with the design, production, execution, or distribution of the Challenge or their immediate family (spouse, parents or step-parents, siblings and step-siblings and children and step-children).

(8) Federal grantees may not use Federal funds to develop COMPETES Act challenge applications unless consistent with the purpose of their grant award.

(9) Federal contractors may not use Federal funds from a contract to develop COMPETES Act challenge applications or to fund efforts in support of a COMPETES Act challenge submission.

(10) CSR reserves the right to cancel, suspend, modify the challenge and/or not award a prize if no submissions are deemed worthy.Start Printed Page 25605

(11) CSR will claim no rights to intellectual property. By participating in this challenge, participant grants to CSR an irrevocable, paid-up, royalty-free, nonexclusive worldwide license to post, link to, share, and display publicly the submission(s) on the Web, newsletters or pamphlets, and other information products such as a future Funding Opportunity Announcement or other study to develop the methodology. In addition, CSR may incorporate proposed ideas into a future Request for Applications (RFA), Request for Proposals (RFP) or an implemented study to develop the methodology, but an award of a prize does not guarantee the proposed idea will be implemented.

(12) By participating in this challenge, participant agrees that the submission is participant's original work and that all proposed ideas are participant's original effort. It is the responsibility of the participant to obtain any rights necessary to use, disclose, or reproduce any intellectual property owned by third parties and incorporated in the entry for all anticipated uses of the submission. Submissions must not violate or infringe upon the rights of other parties, including, but not limited to, privacy, publicity or intellectual property rights, or material that constitutes copyright or license infringement.

(13) By participating in this challenge, each participant (including each individual making up a group participant) agrees to assume any and all risks and waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from participation in this prize challenge, whether the injury, death, damage, or loss arises through negligence or otherwise.

(14) Based on the subject matter of the challenge, the type of work that it will possibly require, as well as an analysis of the likelihood of any claims for death, bodily injury, or property damage, or loss potentially resulting from challenge participation, participants are not required to obtain liability insurance or demonstrate financial responsibility in order to participate in this challenge.

(15) By participating in this challenge, each participant agrees to indemnify the Federal Government against third party claims for damages arising from or related to challenge activities.

(16) An individual shall not be deemed ineligible because the individual used Federal facilities or consulted with Federal employees during this challenge if the facilities and employees are made available to all individuals participating in the challenge on an equitable basis.

(17) In the case of groups, a single, individual group member will submit the submission on behalf of the group and certify that the submission meets all challenge rules.

(18) The decision of the award approving official is final and cannot be contested. The award approving official is the Director of the Center for Scientific Review.

Submission Process for Participants: Participants should submit all entry materials to

Amount of the Prize: CSR may award up to a total of four prizes in two categories: Best Empirically-Based Submission and Most Creative Submission. In each of these two categories, CSR may award a first prize in the amount of $10,000 and a second prize in the amount of $5,000. Each submission is eligible for only one prize (i.e., a single submission cannot win more than one prize for this challenge).

Prizes awarded under this challenge will be paid by electronic funds transfer and may be subject to Federal income taxes. HHS will comply with the Internal Revenue Service withholding and reporting requirements, where applicable. If a group or entity is selected as a winner, CSR will pay the prize to an individual representative of the group or entity designated in the cover letter required as part of the submission. To the extent applicable, it is this individual's responsibility to distribute the prize to group (or entity) members.

Basis Upon Which Submissions Will Be Evaluated: After CSR receives and de-identifies the submissions, the submissions will be evaluated according to a two-stage process: (1) Technical merit will be evaluated for its potential to detect bias in peer review (High, Medium, Low Impact) by a panel of experts in fields relevant to peer review and reviewer bias, and (2) High Impact submissions will be evaluated and rank-ordered based on the judging criteria (see judging criteria below) by a panel of judges comprised of federal employees who will recommend the winning entries.

The final awards will be approved by the Director of the Center for Scientific Review; provided, however, that CSR reserves the right to cancel, suspend, modify the challenge and/or not award a prize if no submissions are deemed worthy.

The judging criteria for the best empirically based and most creative submissions are as follows:

Best Empirically-Based Submission

  • Theoretically based and/or hypothesis driven
  • Proposes an experimental design
  • Well-grounded in peer reviewed empirical literature
  • Proposes measurement methods
  • Feasibility of implementation
  • Related to the NIH Peer Review Process

Most Creative Submission

  • Proposes novel concepts or translates existing concepts in a novel way
  • Challenges existing paradigms
  • The proposed project has potential to be translated for use in an experimental design
  • Creative ways to apply ideas
  • Implementation is feasible
  • Relates to the NIH Peer Review Process

Submission Requirements: This challenge is for the solicitation of ideas for the detection of bias in NIH Peer Review, therefore a full development of new measures is not required. The following materials must be uploaded to or sent in hardcopy to the Office of the Director, Attention: Denise McGarrell, Center for Scientific Review, 6701 Rockledge Drive, Suite 3030, Bethesda, Maryland 20892 by the deadline. Incomplete submissions will not be considered. All submissions must be written in English.

  • Cover sheet with title of the submission and the participant's name or names of group members and contact information. In the case of groups (and entities), indicate one group member responsible for corresponding with CSR. Also indicate which group member will be responsible for receiving the prize for distribution, as applicable, among group members.
  • Challenge submission documents. Note: The 2-page challenge idea should be anonymous (i.e., not include identifying information of the participant). Submissions shall not exceed 2 single-spaced pages (not to include cover page, references or parental consent document, if applicable) and shall be constrained to no less than one inch margins and 11 pt. Ariel font. All submissions must be submitted in .doc (Word) format. Submissions should include the following sections:

Aims: Describe the goals for your proposed approach to the detection of bias in peer review and the anticipated outcomes.Start Printed Page 25606

Approach: Provide a detailed description of your proposed methods and procedures. Describe how you might measure the effectiveness of your plan in accomplishing your proposed aims.

Implementation: Explain how your method might be implemented as part of NIH Peer Review. Include how your proposed method might be tested and, if effective, how it might be disseminated across the NIH.

  • As applicable, the signed Parental Consent Document.
  • Submissions not conforming to these specifications will be disqualified.


Ginther DK et al. (2011). Race, ethnicity, and NIH research awards. Science, 333 (1015-1019).

Ginther DK, Haak LL, Schaffer WT, & Kington R. (2012). Are race, ethnicity, and medical school affiliation associated with NIH R01 type 1 award probability for physician investigators? Academic Medicine, 87 (11), 1516-1524.

Start Signature

Dated: April 29, 2014.

Richard Nakamura,

Director, Center for Scientific Review, National Institutes of Health.

End Signature End Supplemental Information

[FR Doc. 2014-10196 Filed 5-2-14; 8:45 am]