Assessing scientists for hiring, promotion, and tenure

PLoS Biology

Volumn 16, Issue 3, 2018, Pages

  Moher, David ^a,b   Naudet, Florian ^b,c   Cristea, Ioana A ^b,d   Miedema, Frank ^e   Ioannidis, John P A ^b,f,g   Goodman, Steven N ^b,f,g  

^a OTTAWA HOSPITAL RESEARCH INSTITUTE   (Canada)

^b STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^c UNIVERSITÉ DE RENNES 1   (France)

^d BABE BOLYAI UNIVERSITY   (Romania)

^e UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^f Department of Medicine ^*  (United States)

^g STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; BIOMEDICINE; DISTRICT OF COLUMBIA; HUMAN; LEADERSHIP; PLANT SEED; SCIENTIST; LABORATORY PERSONNEL; PERSONNEL MANAGEMENT; REPRODUCIBILITY; RESEARCH; STANDARDS; UNIVERSITY;

EMPLOYEE INCENTIVE PLANS; FACULTY; HUMANS; LABORATORY PERSONNEL; REPRODUCIBILITY OF RESULTS; RESEARCH;

EID: 85045249950     PISSN: 15449173     EISSN: 15457885     Source Type: Journal    
DOI: 10.1371/journal.pbio.2004089     Document Type: Article

References (77)

1. Hammarfelt B, Recognition and reward in the academy: valuing publication oeuvres in biomedicine, economics and history. Aslib J Inform Manag2017; 69(5):607–23.
2. Quan W, Chen B, Shu F. Publish Or Impoverish: An Investigation Of The Monetary Reward System Of Science In China (1999–2016).[Internet]. Available from: https://arxiv.org/ftp/arxiv/papers/1707/1707.01162.pdf. Last accessed: 22Feb2018.
3. Harley D, Acord SK, Earl-Novell S, Lawrence S, King CJ,. (2010). Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines. [Internet] UC Berkeley: Center for Studies in Higher Education. Available from: https://cshe.berkeley.edu/publications/assessing-future-landscape-scholarly-communication-exploration-faculty-values-and-needs. Last accessed: 22Feb2018.
4. Walker RL, Sykes L, Hemmelgarn BR, Quan H, Authors' opinions on publication in relation to annual performance assessment. BMC Med Educ2010Mar9;10:21. doi: 10.1186/1472-6920-10-21 20214826
5. Tijdink JK, Schipper K, Bouter LM, Maclaine Pont P, de Jonge J, Smulders YM, How do scientists perceive the current publication culture? A qualitative focus group interview study among Dutch biomedical researchers. BMJ Open2016;6:e008681. doi: 10.1136/bmjopen-2015-008681 26888726
6. Sturmer S, Oeberst A, Trotschel R, Decker O, Early-career researchers’ perceptions of the prevalence of questionable research practices, potential causes, and open science. Soc Psychol2017; 48(6): 365–371.
7. Garfield E, The history and meaning of the journal impact factor. JAMA2006; 295(1):90–3. doi: 10.1001/jama.295.1.90 16391221
8. Brembs B, Button K, Munafò M, Deep impact: unintended consequences of journal rank. Front Hum Neurosci2013;7:article291.
9. Rouleau G, Open Science at an institutional level: an interview with Guy Rouleau. Genome Biol2017Jan20;18(1):14. doi: 10.1186/s13059-017-1152-z 28109193
10. McKiernan EC, Imagining the ‘open’ university: sharing to improve research and education. PLoS Biol2017; 15(10):e1002614. doi: 10.1371/journal.pbio.1002614 29065148
11. Kleinert S, Horton R, How should medical science change?Lancet2014; 383:197–8doi: 10.1016/S0140-6736(13)62678-1 24411649
12. Begley CG, Ellis LM, Drug development: raise standards for preclinical cancer research. Nature2012;483(7391):531–3. doi: 10.1038/483531a 22460880
13. Ioannidis JP, Acknowledging and overcoming nonreproducibility in basic and preclinical research. JAMA2017;317(10):1019–20. doi: 10.1001/jama.2017.0549 28192565
14. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet2014;383(9913):267–76. doi: 10.1016/S0140-6736(13)62228-X 24411647
15. Chan A-W, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet2014; 383(9913):257–66. doi: 10.1016/S0140-6736(13)62296-5 24411650
16. Heckathorn DD, Snowball versus respondent-driven sampling. Sociol Methodol2011; 41(1): 355–366. doi: 10.1111/j.1467-9531.2011.01244.x 22228916
17. Final report summary—ACUMEN (Academic careers understood through measurement and norms).[Internet] Community Research and Development Information Service. European Commission. Available from: http://cordis.europa.eu/result/rcn/157423_en.pdf. Last accessed: 22Feb2018.
18. Amsterdam call for action on open science.[Internet] The Netherlands EU Presidency 2016. Available from: https://f-origin.hypotheses.org/wp-content/blogs.dir/1244/files/2016/06/amsterdam-call-for-action-on-open-science.pdf. Last Accessed: 22Feb2018.
19. American Society for Cell Biology. DORA. Declaration on Research Assessment. [Internet] Available from: http://www.ascb.org/dora/. Last accessed: 22Feb2018.
20. Hicks D, Wouters P, Waltman L, de Rijcke S, Rafols I, Bibliometrics: The Leiden Manifesto for research metrics. Nature2015;520(7548):429–31. doi: 10.1038/520429a 25903611
21. Wilsdon J, The metric tide: Report of the independent review of the role of metrics in research assessment and management.[Internet] University of Sussex. 2015. Available from: blogs.lse.ac.uk/impactofsocialsciences/files/2015/07/2015_metrictide.pdf. Last accessed: 22Feb2018.
22. Alberts B, Cicerone RJ, Fienberg SE, Kamb A, McNutt M, Nerem RM, et al. Scientific Integrity. Self-correction in science at work. Science2015;348(6242):1420–2. doi: 10.1126/science.aab3847 26113701
23. The culture of scientific research in the UK. Nuffield Council on Bioethics. [Internet] Available from: http://nuffieldbioethics.org/wp-content/uploads/Nuffield_research_culture_full_report_web.pdf. Last Accessed: 22Feb2018.
24. Panel criteria and working methods.[Internet] [REF 2014/REF 01.2012.] Available from: https://www.imperial.ac.uk/media/imperial-college/research-and-innovation/public/Main-panel-criteria.pdf; http://www.ref.ac.uk/2014/media/ref/content/pub/REF%20Brief%20Guide%202014.pdf. Last Accessed: 22Feb2018.
25. Benedictus R, Miedema F, Fewer numbers, better science. Nature2016;538(7626):453–5. doi: 10.1038/538453a 27786219
26. Edwards MA, Roy S, Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environ Eng Sci2017. 34, 51. doi: 10.1089/ees.2016.0223 28115824
27. Ioannidis JPA, How to make more published research true. PLoS Med2014; 11(10): e1001747. doi: 10.1371/journal.pmed.1001747 25334033
28. Mazumdar M, Messinger S, Finkelstein DM, Goldberg JD, Lindsell CJ, Morton SC, et al. Evaluating academic scientists collaborating in team-based research: A proposed framework. Acad Med2015;90(10):1302–8. doi: 10.1097/ACM.0000000000000759 25993282
29. Ioannidis JP, Khoury MJ, Assessing value in biomedical research: the PQRST of appraisal and reward. JAMA2014;312:483–4. doi: 10.1001/jama.2014.6932 24911291
30. Nosek BA, Spies JR, Motyl M, Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect Psychol Sci2012;7(6):615–31. doi: 10.1177/1745691612459058 26168121
31. Schekman R, Patterson M, Reforming research assessment. eLife2013;2:e00855. doi: 10.7554/eLife.00855 23700504
32. Time to remodel the journal impact factor. Nature2016;535(7613):466. doi: 10.1038/535466a 27466089.
33. Hutchins BI, Yuan X, Anderson JM, Santangelo GM, Relative Citation Ratio (RCR): A new metric that uses citation rates to measure influence at the article level. PLoS Biol2016; 14(9): e1002541. doi: 10.1371/journal.pbio.1002541 27599104
34. Larivière V, Kiermer V, MacCallum CJ, McNutt M, Patterson M, Pulverer B, et al. A simple proposal for the publication of journal citation distributions. bioRxiv. Available from: https://www.biorxiv.org/content/biorxiv/early/2016/09/11/062109.full.pdf. Last accessed: 22Feb2018.
35. Cantor M, Gero S, The missing metric: quantifying contributions of reviewers. R Soc Open Sci2015; 2 (2): 140540. doi: 10.1098/rsos.140540 26064609
36. Olfson M, Wall MM, Blanco C, Incentivizing data sharing and collaboration in medical research-the S-Index. JAMA Psychiatry2017;74(1):5–6. doi: 10.1001/jamapsychiatry.2016.2610 27784040
37. Moher D, Goodman SN, Ioannidis JP, Academic criteria for appointment, promotion and rewards in medical research: where’s the evidence?Eur J Clin Invest2016; 46(5):383–5. doi: 10.1111/eci.12612 26924551
38. Brookshire B. Blame bad science incentives for bad science. [Internet] https://www.sciencenews.org/blog/scicurious/blame-bad-incentives-bad-science. Last accessed: 22Feb2018.
39. Johnson B. The road to the responsible research metrics forum. Higher education funding council for England.[Internet] Available from: http://blog.hefce.ac.uk/2017/03/24/the-road-to-the-responsible-research-metrics-forum/. Last Accessed 22Feb2018.
40. Imperial College London signs DORA. [Internet] Available from: http://www3.imperial.ac.uk/newsandeventspggrp/imperialcollege/newssummary/news_8-2-2017-12-28-7. Last accessed: 22Feb2018.
41. Gadd E. When are journal metrics useful? A balanced call for the contextualized and transparent use of all publication metrics. [Internet] LSE Impact Blog. Available from: http://blogs.lse.ac.uk/impactofsocialsciences/2015/11/05/when-are-journal-metrics-useful-dora-leiden-manifesto/ Last accessed: 22Feb2018.
42. Birkbeck signs San Francisco Declaration on Research Assessment. [Internet] Available from: http://tagteam.harvard.edu/hub_feeds/3649/feed_items/2224509 Last accessed: 22Feb2018.
43. Terama E, Smallman M, Lock SJ, Johnson C, Austwick MZ, Beyond Academia -Interrogating Research Impact in the Research Excellence Framework. PLoS One2016;11(12):e0168533. doi: 10.1371/journal.pone.0168533 27997599
44. Sayer D. Five reasons why the REF is not fit for purpose. [Internet] The Gaurdian 2014 15 Dec. Available from: https://www.theguardian.com/higher-education-network/2014/dec/15/research-excellence-framework-five-reasons-not-fit-for-purpose Last accessed: 22Feb2018.
45. Public Library of Science. PLOS and DORA. [Internet]. Available at: https://www.plos.org/dora. Last Accessed: 15-Feb-2018.
46. Burley, R. BioMed Central and SpringerOpen sign the San Francisco Declaration on Research Assessment. [Internet]. Available at: http://blogs.biomedcentral.com/bmcblog/2017/04/26/biomed-central-and-springeropen-sign-the-san-francisco-declaration-on-research-assessment/. Last Accessed: 14-Feb-2018.
47. Bastian H, Bias in Open Science Advocacy: The Case of Article Badges for Data Sharing. PLOS Blogs. Posted August29, 2017. [Internet] Available from: http://bit.ly/2jq4eR6. Last accessed: 22Feb2018.
48. Kidwell MC, Lazarevi LB, Baranski E, Hardwicke TE, Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biol2016: 14(5): e1002456. doi: 10.1371/journal.pbio.1002456 27171007
49. Nosek BA, Lakens D, Registered reports: a method to increase the credibility of published results. Soc Psychol2014; 45: 137–41.
50. Ioannidis JPA, Boyack K, Wouters PF, Citation Metrics: A primer on how (not) to normalize. PLoS Biol2016; 14(9): e1002542. doi: 10.1371/journal.pbio.1002542 27599158
51. Janssens ACJW, Goodman M, Powell KR, Gwinn M, A critical evaluation of the algorithm behind the Relative Citation Ratio (RCR). PLoS Biol2017; 15(10): e2002536. doi: 10.1371/journal.pbio.2002536 28968388
52. Ioannidis JP, Klavans R, Boyack KW, Multiple citation indicators and their composite across scientific disciplines. PLoS Biol2016; 14(7): e1002501. doi: 10.1371/journal.pbio.1002501 27367269
53. Boyer L, Scholarship reconsidered: priorities of the professoriate. Lawrenceville, NJ: Princeton University Press, 1990. 152p.
54. Lapinski S, Piwowar H, Priem J, Riding the crest of the altmetrics wave: How librarians can help prepare faculty for the next generation of research impact metrics. Coll Res Libraries News2013; 74(6): 292–300.
55. Zare RN, Assessing academic researchers. Angew Chem Int Ed Engl2012;51(30):7338–9. doi: 10.1002/anie.201201011 22513978
56. Ovseiko PV, Chapple A, Edmunds LD, Ziebland S, Advancing gender equality through the Athena SWAN Charter for Women in Science: an exploratory study of women's and men's perceptions. Health Res Policy Syst2017;15(1):12. doi: 10.1186/s12961-017-0177-9 28222735
57. CWTS Leiden Ranking. Responsible use. [Internet] Available from http://www.leidenranking.com/information/responsibleuse. Last Accessed 22Feb2018.
58. Pasterkamp G, Hoefer I, Prakken B, Lost in citation valley. Nat Biotechnol2016;34(10):1016–1018. doi: 10.1038/nbt.3691 27727210
59. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, et al. Promoting an open research culture. Science2015; 348(6242):1422–1425. doi: 10.1126/science.aab2374 26113702.
60. Ioannidis JPA, Defending biomedical science in an era of threatened funding. JAMA2017;317(24):2483–2484. doi: 10.1001/jama.2017.5811 28459974
61. Powell-Smith A, Goldacre B, The TrialsTracker: automated ongoing monitoring of failure to share clinical trial results by all major companies and research institutions. F1000Res2016;5:2629doi: 10.12688/f1000research.10010.1 28105310
62. Coens C, Bogaerts J, Collette L, Comment on the “TrialsTracker: Automated ongoing monitoring of failure to share clinical trial results by all major companies and research institutions.” [version 1; referees: 1 approved, 1 approved with reservations]F1000 Res2017;6:71.
63. Flier J, Faculty promotion must assess reproducibility. Nature2017;549(7671):133. doi: 10.1038/549133a 28905925
64. Mogil JS, MacLeod MR, No publication without confirmation. Nature2017;542(7642):409–411. doi: 10.1038/542409a 28230138
65. Topol EJ, Money back guarantees for non-reproducible results?BMJ2016;353:i2770. doi: 10.1136/bmj.i2770 27221803
66. Terämä E, Smallman M, Lock SJ, Johnson C, Austwick MZ, Beyond Academia–Interrogating Research Impact in the Research Excellence Framework. PLoS ONE2016; 11(12): e0168533. doi: 10.1371/journal.pone.0168533 27997599
67. Manville C, Jones MM, Frearson M, Castle-Clarke S, Henham M, Gunashekar S, et al. Preparing impact submissions for REF 2014: An evaluation: Findings and observations. [Internet] Santa Monica, CA: RAND Corporation, 2015. https://www.rand.org/pubs/research_reports/RR727.html. Last Accessed: 22Feb2017.
68. Gaind N, Few UK universities have adopted rules against impact-factor abuse. [Internet] Nature2018 Accessed from: https://www.nature.com/articles/d41586-018-01874-w. Last accessed 14Feb2018.
69. Piwowar H, Priem J, Lariviere V, Alperin JP, Matthias L, Norlander B, et al. The state of OA: a large scale analysis of the prevalence and impact of open access articles.[Internet] PeerJ Preprints Available from: https://doi.org/10.7287/peerj.preprints.3119v1 Last accessed: 22Feb2018.
70. Odell J, Coates H, Palmer K, Rewarding open access scholarship in promotion and tenure: driving institutional change. C&RL News2016;77:7.
71. Assessing current practices in the review, promotion and tenure process. [Internet] Available from: https://publishing.sfu.ca/7297-review-promotion-tenure-project/. Last Accessed: 22Feb2018.
72. Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ, The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ2015;350:h391. doi: 10.1136/bmj.h391 25662947
73. Kontopantelis E, Doran T, Springate DA, Buchan I, Reeves D, Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ2015;350:h2750. doi: 10.1136/bmj.h2750 26058820
74. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev2012;(6):CD000259. doi: 10.1002/14651858.CD000259.pub3 22696318
75. Taylor M, What impact does research have? [Internet] BMJ Opinion. Available from: http://blogs.bmj.com/bmj/2017/05/10/mark-taylor-what-impact-does-our-research-have/. Last Accessed 22Feb18.
76. Biagioli M, Watch out for cheats in citation game. Nature2016;535(7611):201. doi: 10.1038/535201a 27411599
77. Piwowar HA, Day RS, Fridsma DB, Sharing Detailed Research Data Is Associated with Increased Citation Rate. PLoS ONE2007;2(3): e308. doi: 10.1371/journal.pone.0000308 17375194