Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities

ACS Catalysis

Volumn 6, Issue 4, 2016, Pages 2590-2602

  Bligaard, Thomas ^a   Bullock, R Morris ^b   Campbell, Charles T ^c   Chen, Jingguang G ^d,e   Gates, Bruce C ^f   Gorte, Raymond J ^g   Jones, Christopher W ^h   Jones, William D ⁱ   Kitchin, John R ^j   Scott, Susannah L ^k  

^a SLAC NATIONAL ACCELERATOR LABORATORY   (United States)

^b PACIFIC NORTHWEST NATIONAL LABORATORY   (United States)

^c UNIVERSITY OF WASHINGTON   (United States)

^d Columbia University ^*  (United States)

^e BROOKHAVEN NATIONAL LABORATORY   (United States)

^f University of California Davis   (United States)

^g UNIVERSITY OF PENNSYLVANIA   (United States)

^h Georgia Institute of Technology   (United States)

ⁱ UNIVERSITY OF ROCHESTER   (United States)

^j CARNEGIE MELLON UNIVERSITY   (United States)

^k University of California   (United States)

more..

Author keywords

benchmarking; catalytic performance; computational catalysis; electrocatalysis; heterogeneous catalysis; molecular catalysis

Indexed keywords

CATALYSIS; CATALYST ACTIVITY; CATALYST SELECTIVITY; CATALYSTS; ELECTROCATALYSIS;

BEST PRACTICES; CATALYST PERFORMANCE; CATALYTIC PERFORMANCE; COMMUNITY-BASED; COMPLEX REACTION SYSTEMS; ITS APPLICATIONS; MODEL SYSTEM; RESEARCH DATA;

BENCHMARKING;

EID: 84963599520     PISSN: None     EISSN: 21555435     Source Type: Journal    
DOI: 10.1021/acscatal.6b00183     Document Type: Review

References (96)

1. "benchmark, v.". In OED Online [Online] December 2015 ed.; Oxford University Press. http://www.oed.com/view/Entry/251392?result=2&rskey=fbkKBY& (accessed January 17, 2016).
2. Sim, S. E., Easterbrook, S., and Holt, R. C. Using Benchmarking to Advance Research: A Challenge to Software Engineering, 25th International Conference on Software Engineering (ICSE '03), Washington, DC, IEEE Computer Society: Washington, DC, 2003; pp 74-83.
3. Venema, V. Variable Variability. http://variable-variability.blogspot.com/2012/01/what-distinguishes-benchmark.html (accessed January 17, 2016).
4. Farlin, J.; Majewsky, M. Environ. Sci. Technol. 2015, 49, 2604-2605 10.1021/acs.est.5b00389
5. Schwab, M.; Karrenbach, N.; Claerbout, J. Comput. Sci. Eng. 2000, 2, 61-67 10.1109/5992.881708
6. Buckheit, J.; Donoho, D. L. In Wavelets and Statistics, Antoniadis, A., Ed.; Springer: New York, 1995; pp 55-81.
7. Donoho, D. L. Biostatistics 2010, 11, 385-388 10.1093/biostatistics/kxq028
8. Piwowar, H. A.; Day, R. S.; Fridsma, D. B. PLoS One 2007, 2, e308 10.1371/journal.pone.0000308
9. Wells, P. B. Platinum Met. Rev. 1985, 29, 168-174
10. Bond, G. C. In Handbook of Heterogeneous Catalysis, Ertl, G.; Knözinger, H.; Schüth, F.; Weitkamp, J., Eds. Wiley-VCH: Weinheim, Germany, 2008; Vol. 1, pp 700-715.
11. World Gold Council Gold Bull. 2003, 36, 24 10.1007/BF03214864
12. Bond, G. C.; Sing, K. S. W.; Everett, D. H.; Haul, R. A. W.; Moscou, L.; Pierotti, R. A.; Rouquerol, J.; Siemieniewska, T. In Handbook of Heterogeneous Catalysis, Part A, Ertl, G.; Knözinger, H.; Weitkamp, J., Eds.; Wiley-VCH Verlag GmbH: Weinheim, Germany, 1997; pp 1489-1558.
13. Hammond, K. D.; Conner, W. C., Jr. Adv. Catal. 2013, 56, 1-101 10.1016/B978-0-12-420173-6.00001-2
14. Alexeev, O.; Kim, D.-W.; Graham, G. W.; Shelef, M.; Gates, B. C. J. Catal. 1999, 185, 170-181 10.1006/jcat.1999.2489
15. Kaminsky, W. J. Chem. Soc., Dalton Trans. 1998, 1413-1418 10.1039/a800056e
16. Resconi, L.; Cavallo, L.; Fait, A.; Piemontesi, F. Chem. Rev. 2000, 100, 1253-1345 10.1021/cr9804691
17. Doyle, M. P.; Forbes, D. C. Chem. Rev. 1998, 98, 911-936 10.1021/cr940066a
18. Alsters, P. L.; Witte, P. T.; Neumann, R.; Sloboda Rozner, D.; Adam, W.; Zhang, R.; Reedijk, J.; Gamez, P.; ten Elshof, J. E.; Chowdhury, S. R. In Mechanisms in Homogeneous and Heterogeneous Epoxidation Catalysis; Oyama, S. T., Ed.; Elsevier: Amsterdam, 2008; pp 415-428.
19. Artero, V.; Savéant, J.-M. Energy Environ. Sci. 2014, 7, 3808-3814 10.1039/C4EE01709A
20. Costentin, C.; Savéant, J.-M. ChemElectroChem 2014, 1, 1226-1236 10.1002/celc.201300263
21. Costentin, C.; Passard, G.; Savéant, J.-M. J. Am. Chem. Soc. 2015, 137, 5461-6467 10.1021/jacs.5b00914
22. Gasteiger, H. A.; Kocha, S. S.; Sompalli, B.; Wagner, F. T. Appl. Catal., B 2005, 56, 9-35 10.1016/j.apcatb.2004.06.021
23. Frydendal, R.; Paoli, E. A.; Knudsen, B. P.; Wickman, B.; Malacrida, P.; Stephens, I. E. L.; Chorkendorff, I. ChemElectroChem 2014, 1, 2075-2081 10.1002/celc.201402262
24. Colic, V.; Tymoczko, J.; Maljusch, A.; Ganassin, A.; Schuhmann, W.; Bandarenka, A. S. ChemElectroChem 2015, 2, 143-149 10.1002/celc.201402295
25. McCrory, C. C. L.; Jung, S.; Ferrer, I. M.; Chatman, S. M.; Peters, J. C.; Jaramillo, T. F. J. Am. Chem. Soc. 2015, 137, 4347-4357 10.1021/ja510442p
26. McCrory, C. C. L.; Jung, S.; Peters, J. C.; Jaramillo, T. F. J. Am. Chem. Soc. 2013, 135, 16977-16987 10.1021/ja407115p
27. Steinmann, S. N.; Corminboeuf, C. J. Chem. Theory Comput. 2011, 7, 3567-3577 10.1021/ct200602x
28. Marom, N.; Tkatchenko, A.; Rossi, M.; Gobre, V. V.; Hod, O.; Scheffler, M.; Kronik, L. J. Chem. Theory Comput. 2011, 7, 3944-3951 10.1021/ct2005616
29. Santra, B.; Michaelidis, A.; Scheffler, M. J. Chem. Phys. 2007, 127, 184104 10.1063/1.2790009
30. Xu, X.; Zhang, W.; Tang, M.; Truhlar, D. G. J. Chem. Theory Comput. 2015, 11, 2036-2052 10.1021/acs.jctc.5b00081
31. Korth, M.; Grimme, S. J. Chem. Theory Comput. 2009, 5, 993-1003 10.1021/ct800511q
32. Zhao, Y.; Truhlar, D. G. J. Chem. Theory Comput. 2009, 5, 324-333 10.1021/ct800386d
33. Chen, S.; Raugei, S.; Rousseau, R.; Dupuis, M.; Bullock, R. M. J. Phys. Chem. A 2010, 114, 12716-12724 10.1021/jp106800n
34. Mollenhauer, D.; Gaston, N. J. Comput. Chem. 2014, 35, 986-997 10.1002/jcc.23578
35. Guner, V.; Khuong, K. S.; Leach, A. G.; Lee, P. S.; Bartberger, M. D.; Houk, K. N. J. Phys. Chem. A 2003, 107, 11445-11459 10.1021/jp035501w
36. Keith, J. A.; Carter, E. A. J. Chem. Theory Comput. 2012, 8, 3187-3206 10.1021/ct300295g
37. Yang, K.; Zheng, J.; Zhao, Y.; Truhlar, D. G. J. Chem. Phys. 2010, 132, 164117 10.1063/1.3382342
38. Bhattacharya, S.; Levchenko, S. V.; Ghiringhelli, L. M.; Scheffler, M. Phys. Rev. Lett. 2013, 111, 135501 10.1103/PhysRevLett.111.135501
39. Paz-Borbón, L. O.; Barcaro, G.; Fortunelli, A.; Levchenko, S. V. Phys. Rev. B: Condens. Matter Mater. Phys. 2012, 85, 155409 10.1103/PhysRevB.85.155409
40. Ghiringhelli, L. M.; Gruene, P.; Lyon, J. T.; Rayner, D. M.; Meijer, G.; Fielicke, A.; Scheffler, M. New J. Phys. 2013, 15, 083003 10.1088/1367-2630/15/8/083003
41. Wellendorff, J.; Silbaugh, T. L.; Garcia-Pintos, D.; Nørskov, J. K.; Bligaard, T.; Studt, F.; Campbell, C. T. Surf. Sci. 2015, 640, 36-44 10.1016/j.susc.2015.03.023
42. Kroes, G.-J. J. Phys. Chem. Lett. 2015, 6, 4106-4114 10.1021/acs.jpclett.5b01344
43. Hwang, T.; Goldsmith, B. R.; Peters, B.; Scott, S. L. Inorg. Chem. 2013, 52, 13904-13917 10.1021/ic401343m
44. Goldsmith, B. R.; Hwang, T.; Seritan, S.; Peters, B.; Scott, S. L. J. Am. Chem. Soc. 2015, 137, 9604-9616 10.1021/jacs.5b03750
45. Plata, R. E.; Singleton, D. A. J. Am. Chem. Soc. 2015, 137, 3811-3826 10.1021/ja5111392
46. Campbell, C. T. J. Catal. 2001, 204, 520-524 10.1006/jcat.2001.3396
47. Stegelmann, C.; Andreasen, A.; Campbell, C. T. J. Am. Chem. Soc. 2009, 131, 8077-8082 10.1021/ja9000097
48. Kitchin, J. R. ACS Catal. 2015, 5, 3894-3899 10.1021/acscatal.5b00538
49. Aalbersberg, I. J.; Atzeni, S.; Koers, H.; Specker, B.; Zudilova-Seinstra, E. Liber Quart. 2014, 23, 274-299 10.18352/lq.8446
50. Peterson, K. A.; Feller, D.; Dixon, D. A. Theor. Chem. Acc. 2012, 131, 1079 10.1007/s00214-011-1079-5
51. Rohmann, K.; Hölscher, M.; Leitner, W. J. Am. Chem. Soc. 2016, 138, 433-443 10.1021/jacs.5b11997
52. Jensen, J. H. Phys. Chem. Chem. Phys. 2015, 17, 12441-12451 10.1039/C5CP00628G
53. Jones, G.; Bligaard, T.; Abild-Pedersen, F.; Nørskov, J. K. J. Phys.: Condens. Matter 2008, 20, 064239 10.1088/0953-8984/20/6/064239
54. Norskøv, J. K.; Abild-Pedersen, F.; Studt, F.; Bligaard, T. Proc. Natl. Acad. Sci. U. S. A. 2011, 108, 937-943 10.1073/pnas.1006652108
55. Nørskov, J. K.; Bligaard, T.; Rossmeisl, J.; Christensen, C. H. Nat. Chem. 2009, 1, 37-46 10.1038/nchem.121
56. Calle-Vallejo, F.; Loffreda, D.; Koper, M. T. M.; Sautet, P. Nat. Chem. 2015, 7, 403-410 10.1038/nchem.2226
57. Rodriguez, J. A.; Ma, S.; Liu, P.; Hrbek, J.; Evans, J.; Perez, M. Science 2007, 318, 1757-1760 10.1126/science.1150038
58. Cargnello, M.; Doan-Nguyen, V.; Gordon, T. R.; Diaz, R. E.; Stach, E. A.; Gorte, R. J.; Fornasiero, P.; Murray, C. B. Science 2013, 341, 771-773 10.1126/science.1240148
59. Hansen, T. W.; Wagner, J. B. ACS Catal. 2014, 4, 1673-1685 10.1021/cs401148d
60. Su, D. S.; Zhang, B.; Schlögl, R. Chem. Rev. 2015, 115, 2818-2882 10.1021/cr500084c
61. Madon, R. J.; Boudart, M. Ind. Eng. Chem. Fundam. 1982, 21, 438-447 10.1021/i100008a022
62. Davis, M. E.; Davis, R. J. Fundamentals of Chemical Reaction Engineering; McGraw-Hill: Boston, 2003; pp 184-232.
63. Canivet, J.; Vandichel, M.; Farrusseng, D. Dalton Trans. 2016, 45, 4090-4099 10.1039/C5DT03522H
64. Kozuch, S.; Martin, J. M. L. ACS Catal. 2012, 2, 2787-2794 10.1021/cs3005264
65. Lente, G. ACS Catal. 2013, 3, 381-382 10.1021/cs300846b
66. Blackmond, D. G. Angew. Chem., Int. Ed. 2005, 44, 4302-4320 10.1002/anie.200462544
67. Jones, C. W.; Smith, D. J.; Gunnoe, T. B.; Zhao, H.; Sautet, P.; Scott, S. L.; Xu, B.-Q. ACS Catal. 2014, 4, 2827-2828 10.1021/cs501043q
68. Jones, C. W. Top. Catal. 2010, 53, 942-952 10.1007/s11244-010-9513-9
69. Hintermair, U.; Franció, G.; Leitner, W. Chem. Commun. 2011, 47, 3691-3701 10.1039/c0cc04958a
70. Hickman, A. J.; Villalobos, J. M.; Sanford, M. S. Organometallics 2009, 28, 5316-5322 10.1021/om900495n
71. Joint Center for Artificial Photosynthesis (JCAP). ElectrocatalystBenchmarking Database. https://internal.solarfuelshub.org/jcapresources/benchmarking/catalysts/ (accessed January 17, 2016).
72. Chirik, P. J. Organometallics 2015, 34, 4783 10.1021/acs.organomet.5b00815
73. Prinz, F.; Schlange, T.; Asadullah, K. Nat. Rev. Drug Discovery 2011, 10, 712-713 10.1038/nrd3439-c1
74. Begley, C. G.; Ellis, L. M. Nature 2012, 483, 531-533 10.1038/483531a
75. Ioannidis, J. P. A. PLoS Med. 2005, 2, e124 10.1371/journal.pmed.0020124
76. Shanks, D. R.; Newell, B. R.; Lee, E. H.; Balakrishnan, D.; Ekelund, L.; Cenac, Z.; Kavvadia, F.; Moore, C. PLoS One 2013, 8, e56515 10.1371/journal.pone.0056515
77. Open Science Collaboration. Science 2015, 349, 943.
78. Anonymous Unreliable Research: Trouble at the Lab. The Economist; October 19, 2013; pp 26-30.
79. Lehrer, J. The Truth Wears Off: Is There Something Wrong with the Scientific Method? New Yorker; December 13, 2010; pp 52-57;.
80. Freedman, D. H. Lies, Damned Lies, and Medical Science. Atlantic Monthly; November, 2010.
81. Carey, B. Science, Now Under Scrutiny Itself. New York Times; June 15, 2015.
82. Freedman, L. P.; Cockburn, I. M.; Simcoe, T. S. PLoS Biol. 2015, 13, e1002165 10.1371/journal.pbio.1002165
83. Fecher, B.; Friesike, S.; Hebing, M. PLoS One 2015, 10, e0118053 10.1371/journal.pone.0118053
84. Sweedler, J. V. Anal. Chem. 2015, 87, 11603-11604 10.1021/acs.analchem.5b04300
85. Sharp, P. A.; Kleppner, D. Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age; Institute of Medicine and National Academy of Sciences. The National Academies Press: Washington, D.C., 2009; http://www.nap.edu/catalog/12615/ensuring-the-integrity-accessibility-and-stewardship-of-research-data-in-the-digital-age (accessed January 17, 2016).
86. Molloy, J. C. PLoS Biol. 2011, 9, e1001195 10.1371/journal.pbio.1001195
87. Anonymous Nat. Methods 2013, 10, 367 10.1038/nmeth.2471
88. Transparency and Openness Promotion Committee. https://cos.io/top/ (accessed January 17, 2016).
89. Nosek, B. A.; Alter, G.; Banks, G. C.; Borsboom, D.; Bowman, S. D.; Breckler, S. J.; Buck, S.; Chambers, C. D.; Chin, G.; Christensen, G.; Contestabile, M.; Dafoe, A.; Eich, E.; Freese, J.; Glennerster, R.; Goroff, D.; Green, D. P.; Hesse, B.; Humphreys, M.; Ishiyama, J.; Karlan, D.; Kraut, A.; Lupia, A.; Mabry, P.; Madon, T.; Malhotra, N.; Mayo-Wilson, E.; McNutt, M.; Miguel, E.; Paluck, E. L.; Simonsohn, U.; Soderberg, C.; Spellman, B. A.; Turitto, J.; VandenBos, G.; Vazire, S.; Wagenmakers, E. J.; Wilson, R.; Yarkoni, T. Science 2015, 348, 1422-1425 10.1126/science.aab2374
90. Alberts, B.; Cicerone, R. J.; Fienberg, S. E.; Kamb, A.; McNutt, M.; Nerem, R. M.; Schekman, R.; Shiffrin, R.; Stodden, V.; Suresh, S.; Zuber, M. T.; Pope, B. K.; Jamieson, K. H. Science 2015, 348, 1420-1422 10.1126/science.aab3847
91. Scientific Integrity and Transparency; 113-10; Committee on Science, Space, and Technology, Subcommittee on Research, U.S. Government Printing Office: Washington, D.C., March 5, 2013; http://www.gpo.gov/fdsys/pkg/CHRG-113hhrg79929/pdf/CHRG-113hhrg79929.pdf (accessed January 17, 2016).
92. Peer review in scientific publications: Eighth Report of Session 2010-2012; HC-856; House of Commons Science and Technology Committee, The Stationary Office, Ltd.: London, U.K., 2011; http://www.parliament.uk/business/committees/committees-a-z/commons-select/science-and-technology-committee/inquiries/peer-review (accessed January 17, 2016).
93. Buck, S. Science 2015, 348, 1403 10.1126/science.aac8041
94. NIST Materials Genome Initiative. https://mgi.nist. gov/density-functional-theory-dft-informatics-and-repositories (accessed January 17, 2016).
95. NoMaD (Novel Materials Discovery) Repository. http://nomad-repository.eu/cms/ (accessed January 17, 2016).
96. Benchmark Energy & Geometry Database (BEGDB). http://www.begdb.com/ (accessed January 17, 2016).