New EMSL Double-blind Pilot Makes Research Proposal Review Anonymous

A certain level of bias, even unconscious bias, can play into the selection of proposals for research at user science facilities. 

Established researchers, for example, can be more likely to have a proposal accepted than that of an early career researcher, said Nikki Powell, deputy of user services at the Environmental Molecular Sciences Laboratory. Powell said the reason for this, even if unintentional, is reviewers may lean into supporting projects whose principal investigators and established work history have a good reputation within their research area. But this can mean that researchers earlier in their career may have a lesser likelihood of their proposal being selected, she said. 

This potential bias, she said, is one of several reasons why EMSL is moving forward with a new pilot where both individuals submitting research proposals, as well as individuals reviewing proposals, are kept anonymous—a process known as double-blind peer review. 

“EMSL has a growing commitment to be a diverse and inclusive user program,” Powell said. “By removing types of bias, we can make the process more equitable, more transparent, and more open and accessible for all.” 

How it works 

Through the double-blind peer review process, neither the proposal submitter, nor the proposal reviewer, are aware of one another’s identities or have access to any identifiable information about previous research experience, background, or associated personnel or organizations. Researchers cannot claim ownership of past work, nor include names of personnel or organizations associated. They must write citations in the form of a number in a square bracket rather than citing a specific published work outright. Additionally, when referencing their own work, they must use neutral, third-person wording. 

On the proposal review panel side of things, in addition to submitting conflict of interest information as is common practice with all EMSL proposal calls, the panel features an individual whose primary purpose is to make sure that conversations don’t stray into speculation of identities, in addition to generally helping keep discussions on track. 

With these parameters in place, Powell said proposals are reviewed and rated purely on scientific content, merit, and mission alignment. 

“These measures will help to maintain the anonymity of the process and create as equitable of a selection process as possible,” she said. 

(Video animation by Stephanie King | Pacific Northwest National Laboratory)

Pilot starting with exploratory call 

EMSL is specifically piloting its new double-blind process with its Call for Exploratory Research Proposals where researchers can submit proposals for research that last up to nine months. Research activities with this type of call must employ certain highlighted capabilities, which are described on the EMSL exploratory research web page, to advance scientific understanding in EMSL’s science areas.

“Our exploratory call seemed like a good opportunity to pilot this double-blind peer review process because it is a smaller call that would provide us with greater capacity to manage the process and determine its success,” Powell said. 

After a successful pilot, Powell said they plan to discuss expanding the double-blind peer review to other calls,potentially making it a part of their standard operating procedure. 

Inspired by NASA 

EMSL staff modeled its double-blind peer review process after NASA’s dual anonymous peer review process that was piloted in 2019 beginning with submissions for use of its Hubble Space Telescope. Daniel Evans, assistant deputy associated administrator for research at NASA, said before implementing their dual anonymous review, they routinely saw a higher proposal acceptance rate for male principal investigators compared to female investigators. 

“This was happening year after year,” he said. “We started to think about unconscious bias and how that played in the review panels of proposals. We decided to switch to a dual anonymous peer review process, which has shown staggering success.” 

Evans said in their first year of the pilot, the gender gap was practically eliminated between male and female researchers. Additionally, they saw a dramatic increase in both the number of proposals they received, as well as the number of proposals selected, by early career researchers. 

Evans has since had conversations with EMSL leadership about EMSL’s new process and is excited that the team is moving forward. 

“While we know that this can’t be the only tool in the toolbox in regard to improving things for equity and diversity, it is a positive step forward in making the science community as diverse as the nation is,” he said. “That is how you produce great science— you have great diverse teams.” 

Next steps and double-blind specifics 

Those interested in submitting a letter of intent for the call should visit the EMSL Call for Exploratory Research Proposals web page to learn about all steps involved. The call for proposals opens Tuesday, June 7. For more information specifically about the double-blind peer review process, visit the Double-Blind Peer Review Pilot web page. 

For the fiscal year 2023 call, individuals will be invited to submit a proposal after a successful letter of intent. There are no changes to the letter of intent submission process—only those invited for a full proposal will need to anonymize their proposal. 

EMSL will host a webinar titled “Understanding the Double-Blind Project Review Process” at noon on Wednesday, June 1, on Zoom, which will go over details about the new process and allow individuals to ask questions. The webinar will feature Powell, Evans, and Davinia Salvachúa, an EMSL user and member of the review panel for the new process. Register to attend. 

To aid in their submission process, researchers are still encouraged to contact or continue discussions with EMSL staff regarding their research goals and resource needs. EMSL staff helping individuals in the proposal process will not be a part of the peer review panel for proposals. 

For more information on EMSL proposal calls, visit emsl.pnnl.gov/proposals or email emsl@pnnl.gov.