Significant improvements in R&R require a significant change in the way that academic research is performed. One such change could be the wide availability of benchmarking standards. A perspective piece by Bligaard et al. has discussed many benefits of establishing benchmarking standards in catalysis and enhancing R&R. Some of these benefits are summarized below:

​

• "According to Donoho, reproducible research confers the following advantages: (1) improved work and work habits; (2) better teamwork; (3) greater impact, resulting from less inadvertent competition and more acknowledgment; and (4) greater continuity and cumulative impact."

• “... benchmarking can clarify the problems to be solved and lead to more rigorous assessment of research results, as well as better assessment methods.”

• “... it promotes research conduct that is collaborative, open, and ethical, thereby promoting community-building”

• “Improved benchmarking has the potential to boost the translation of fundamental catalysis science into the energy technology sector…”

• “... experimental benchmarks will drive improvements in computational accuracy, which will in turn provide deeper understanding that suggests new experiments and more reliable predictions of better catalysts to be tested by experiment.”

• “New researchers could benefit from readily accessible guides about how to best make such measurements…”

​

In spite of it's promise, there are several hurdles to implementing community benchmarking standards in our field. 

​

• "... benchmarking requires communication and collaboration within a community to establish consensus about which questions are valid and how to evaluate their answers."

• "A successful benchmark will be accessible, affordable, clear, relevant, solvable, portable, reproducible, and scalable.“

• "Further benchmarking specifications may be appropriate when the primary objective is comparison within a narrow class of catalytic materials (e.g., metal nanoparticles of different sizes, or on different solid supports). However, they become problematic for comparing disparate kinds of catalytic materials (e.g., molecular catalysts vs supported metal nanoparticles) because of their vastly different operating conditions, or for optimizing more than one performance metric."

• "[Comparing two different catalytic materials] is complicated by two commonly encountered issues: (1) optimum reaction conditions for one catalyst may be markedly suboptimal for another; and (2) the nature and number of the active sites may evolve during the test, complicating normalization of parameters such as rates and yields."

• "Important questions remain about the availability of resources to conduct such work, as well as how to incentivize and recognize efforts that may be regarded as more routine than innovative." 

​

As described above, the challenges to implementing community wide benchmarking standards are significant. Below, we present our ideas for tackling many of these hurdles. 

​

Benchmark Testing Facilities

​

"There is legitimate concern in the scientific community about the additional time commitment and the need to depend on essentially volunteer curation efforts that are difficult to sustain over the long-term, or on commercial operations that may attempt to monetize scientific information created with public funds" - Bligaard et al.

 

University PI's should not be responsible for benchmarking efforts, which do not align with their promotion incentives. However, an ideal academic unit DOES EXIST that can take on these activities: university core facilities. Although prevalent at most universities and in other research fields, to our knowledge, the only current example of this specific type of facility in our field is the Reactor Engineering and Catalyst Testing (REACT) core facility at Northwestern University. Generally, core facilities are user pay-for-service facilities that are well equipped for this type of work. For example, the REACT facility already has the necessary experts, instrumentation, administrative support, financial model, and acknowledgment policies in place to perform this work today. Further, it has the capability of working with users outside of Northwestern (academic, commercial, and national labs). What sets a core facility apart from a PI lab is incentives: PI's are incentivized (through hiring, promotion, and awards) to generate NEW knowledge and understanding (that can be subsequently published in Journals). Core Facilities are incentivized to FACILITATE users in their pursuit of new science. Providing Benchmarking materials to enhance R&R within the field falls in line with these goals.

​

Illustrated below is the general workflow for how a university PI lab would interact with a Benchmarking Facility such as REACT. First, the PI would request a specific material from the lab. This material would have already been synthesized and characterized using standard protocols, and it would have already been tested for catalytic properties under standard conditions (temperatures and pressures) by the benchmarking facility. Presumably, these measurements will have been inline with previous batches synthesized and distributed to other PI’s in the field. The PI receiving the material can then ensure proper operation of their own equipment by testing the material on their own equipment using the standard testing conditions. Further, the PI can then use the material as a benchmark material under their own, specified testing conditions. To further validate these results, the testing facility can test the PI’s materials under the PI’s conditions. When all data has been verified, the experimental data can be stored in a publicly available database, enabling analysis by machine learning.