Assessment of potential bias in research grant peer review in Canada

CMAJ

Volumn 190, Issue 16, 2018, Pages E489-E499

  Tamblyn, Robyn ^a   Girard, Nadyne ^b   Qian, Christina J ^c   Hanley, James ^d  

^a Institute of Health Services and Policy Research   (Canada)

^b CANADIAN INSTITUTES OF HEALTH RESEARCH   (Canada)

^c Departments of Medicine   (Canada)

^d MCGILL UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; AGED; ARTICLE; CANADA; FEMALE; FUNDING; HUMAN; MALE; MEDICAL RESEARCH; MIDDLE AGED; NONLINEAR SYSTEM; PEER REVIEW; SCIENCE; SCIENTIFIC LITERATURE; SCIENTIST; SYSTEMATIC ERROR; FINANCIAL MANAGEMENT; MEDICINE; PERSONNEL; PRODUCTIVITY; STANDARDS; STATISTICAL BIAS; STATISTICS AND NUMERICAL DATA;

ADULT; AGED; BIAS; CANADA; EFFICIENCY; FEMALE; FINANCING, ORGANIZED; HUMANS; MALE; MEDICINE; MIDDLE AGED; PEER REVIEW, RESEARCH; RESEARCH PERSONNEL; RESEARCH SUPPORT AS TOPIC;

EID: 85046026111     PISSN: 08203946     EISSN: 14882329     Source Type: Journal    
DOI: 10.1503/cmaj.170901     Document Type: Article

References (49)

1. Demicheli V, Di Pietrantoni C. Peer review for improving the quality of grant applications. Cochrane Database Syst Rev 2007;(2):MR000003.
2. Mayo NE, Brophy J, Goldberg MS, et al. Peering at peer review revealed high degree of chance associated with funding of grant applications. J Clin Epidemiol 2006;59:842.
3. Graves N, Barnett AG, Clarke P. Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel. BMJ 2011; 343:d4797.
4. Rennie D. Let’s make peer review scientific. Nature 2016;535:31-3.
5. Mervis J. Peering into peer review. Science 2014;343:596-8.
6. Rabesandratana T. The seer of science publishing. Science 2013;342:66-7.
7. Sattler DN, McKnight PE, Naney L, et al. Grant peer review: improving inter-rater reliability with training. PLoS One 2015;10:e0130450.
8. Marsh HW, Jayasinghe UW, Bond N. Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. Am Psychol 2008;63:160-8.
9. Mutz R, Bornmann L, Daniel HD. Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: a general estimating equations approach. PLoS One 2012;7:e48509.
10. Jayasinghe UW, Marsh HW, Bond N. A multilevel cross-classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings. J R Stat Soc Series B Stat Methodol 2003;166:279-300.
11. PierEL,RaclawJ,KaatzA,etal.‘Yourcommentsaremeanerthanyourscore’:score calibration talk influences intra- and inter-panel variability during scientific grant peerreview.ResEval2017;26:1-14.
12. Sheridan J, Savoy JN, Kaatz A, et al. Write more articles, get more grants: the impact of department climate on faculty research productivity. J Womens Health (Larchmt) 2017;26:587-96.
13. Magua W, Zhu X, Bhattacharya A, et al. Are female applicants disadvantaged in National Institutes of Health peer review? Combining algorithmic text mining and qualitative methods to detect evaluative differences in R01 reviewers’ critiques. J Womens Health (Larchmt) 2017;26:560-70.
14. Ginther DK, Kahn S, Schaffer WT. Gender, race/ethnicity, and National Institutes of Health R01 research awards: Is there evidence of a double bind for women of color? Acad Med 2016;91:1098-107.
15. van der Lee R, Ellemers N. Gender contributes to personal research funding success in The Netherlands. Proc Natl Acad Sci U S A 2015;112:12349-53.
16. Jagsi R, DeCastro R, Griffith KA, et al. Similarities and differences in the career trajectories of male and female career development award recipients. Acad Med 2011;86:1415-21.
17. Bornmann L, Mutz R, Daniel H-D. Gender differences in grant peer review: a meta-analysis. J Informetrics 2007;1:226-38.
18. Jayasinghe UW, Marsh HW, Bond N. Peer review in the funding of research in higher education: the Australian experience. Educ Eval Policy Anal 2001;23:343-64.
19. Larivière V, Vignola-Gagné E, Villeneuve C, et al. Sex differences in research funding, productivity and impact: an analysis of Québec university professors. Scientometrics 2011;87:483-98.
20. Gannon F, Quirk S, Guest S. Searching for discrimination: Are women treated fairly in the EMBO postdoctoral fellowship scheme? EMBO Rep 2001;2:655-7.
21. Symonds MR, Gemmell NJ, Braisher TL, et al. Gender differences in publication output: towards an unbiased metric of research performance. PLoS One 2006;1:e127.
22. Kaatz A, Lee YG, Potvien A, et al. Analysis of National Institutes of Health R01 application critiques, impact, and criteria scores: Does the sex of the principal investigator make a difference? AcadMed2016;91:1080-8.
23. Wennerås C, Wold A. Nepotism and sexism in peer-review. Nature 1997;387:341-3.
24. Sandström U, Hällsten M. Persistent nepotism in peer-review. Scientometrics 2008;74:175-89.
25. Eblen MK, Wagner RM, RoyChowdhury D, et al. How criterion scores predict the overall impact score and funding outcomes for National Institutes of Health peer-reviewedapplications.PLoSOne2016;11:e0155060.
26. Ginther DK, Schaffer WT, Schnell J, et al. Race, ethnicity, and NIH research awards. Science2011;333:1015-9.
27. Ginther DK, Haak LL, Schaffer WT, et al. Are race, ethnicity, and medical school affiliation associated with NIH R01 type 1 award probability for physician investigators? AcadMed2012;87:1516-24.
28. Pohlhaus JR, Jiang H, Wagner RM, et al. Sex differences in application, success, and funding rates for NIH extramural programs. Acad Med 2011;86:759-67.
29. Li D. Expertise versus bias in evaluation: evidence from the NIH. Am Econ J Appl Econ 2017;9:60-92.
30. Mutz R, Bornmann L, Daniel H-D. Does gender matter in grant peer review? An empirical investigation using the example of the Austrian science fund. Z Psychol 2012;220:121-9.
31. Women and peer review: an audit of the Wellcome Trust’s decision-making on grants. Report No 8. London (UK): The Wellcome Trust; 1997.
32. Tamblyn R, McMahon M, Girard N, et al. Health services and policy research in the first decade at the Canadian Institutes of Health Research. CMAJ Open 2016;4:E213-21.
33. CIHR internal assessment — report for the 2011 international review. Part 2: CIHR’s budget. Ottawa: Canadian Institutes of Health Research; 2011.
34. Hirsch JE. An index to quantify an individual’s scientific research output. Proc NatlAcadSciUSA2005;102:16569-72.
35. Cicchetti D. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 1994;6:284-90.
36. Nicol MB, Henadeera K, Butler L. NHMRC grant applications: a comparison of “track record” scores allocated by grant assessors with bibliometric analysis of publications. Med J Aust 2007;187:348-52.
37. Kingwell BA, Anderson GP, Duckett SJ, et al.; National Health and Medical Research Council Evaluations and Outcomes Working Committee. Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth. Med J Aust 2006;184:282-6.
38. Bornmann L, Mutz R, Daniel H-D. How to detect indications of potential sources of bias in peer review: a generalized latent variable modeling approach exemplified by a gender study. J Informetrics 2008;2:280-7.
39. Head MG, Fitchett JR, Cooke MK, et al. Differences in research funding for women scientists: a systematic comparison of UK investments in global infectious disease research during 1997–2010. BMJ Open 2013;3:e003362.
40. Kaatz A, Magua W, Zimmerman DR, et al. A quantitative linguistic analysis of National Institutes of Health R01 application critiques from investigators at one institution.AcadMed2015;90:69-75.
41. Ward JE, Donnelly N. Is there gender bias in research fellowships awarded by the NHMRC?MedJAust1998;169:623-4.
42. Moss-Racusin CA, Dovidio JF, Brescoll VL, et al. Science faculty’s subtle gender biases favor male students. Proc Natl Acad Sci U S A 2012;109:16474-9.
43. Abrahamowicz M, Tamblyn RM, Ramsay JO, et al. Detecting and correcting for rater-induced differences in standardized patient tests of clinical competence. Acad Med 1990;65(Suppl):S25-6.
44. Engelhard G Jr, Myford CM. Monitoring faculty consultant performance in the advanced placement English literature and composition program with a many-faceted Rasch model. Report No. 2003-1. College Board Research Report No 2003-1 (ETS RR-03-01). New York: College Entrance Examination Board; 2003.
45. Abdoul H, Perrey C, Tubach F, et al. Non-financial conflicts of interest in academic grant evaluation: a qualitative study of multiple stakeholders in France. PLoS One 2012;7:e35247.
46. Snell RR. Menage a quoi? Optimal number of peer reviewers. PLoS One 2015;10:e0120838.
47. Abdoul H, Perrey C, Amiel P, et al. Peer review of grant applications: criteria used and qualitative study of reviewer practices. PLoS One 2012;7:e46054.
48. Success rates for grant funding. Research Councils UK; 2014.
49. Affiliations: Institute of Health Services and Policy Research (Tamblyn), Canadian Institutes of Health Research, Ottawa, Ont.; Departments of Medicine (Tamblyn) and Epidemiology, and Biostatistics (Girard, Qian, Hanley), Faculty of Medicine, McGill University, Montréal, Que.