Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches

Review of Educational Research

Volumn 79, Issue 3, 2009, Pages 1290-1326

  Marsh, Herbert W ^a   Bornmann, Lutz ^b   Mutz, Rüdiger ^b   Daniel, Hans Dieter ^c   O'Mara, Alison ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b ETH ZURICH   (Switzerland)

^c UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Bias; Gender; Generalizability; Higher education; Meta analysis; Peer review; Substantive methodological synergy; Validity

Indexed keywords

EID: 70449507027     PISSN: 00346543     EISSN: None     Source Type: Journal    
DOI: 10.3102/0034654309334143     Document Type: Review

References (83)

1. Bateman, I.J., & Jones, A.P. (2003). Contrasting conventional with multi-level modelling approaches to meta-analysis: Expectation consistency in UK woodland recreation values. Land Economics, 79, 235-258.
2. Bornmann, L. (2007). Bias cut: Women, it seems, often get a raw deal in science-So how can discrimination be tackled? Nature, 445(7127), 566.
3. Bornmann, L., & Daniel, H.D. (2005). Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of board of trustee decisions. Scientometrics, 63(2), 297-320.
4. Bornmann, L., & Daniel, H.-D. (2007). Gatekeepers of science-Effects of external reviewers' attributes on the assessments of fellowship applications. Journal of Informetrics, 1(1), 83-91.
5. Bornmann, L., & Daniel, H.-D. (2008). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie-International Edition, 47(38), 7173-7178.
6. Bornmann, L., & Mutz, R., & Daniel, H.D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1(3) 226-238.
7. Bornmann, L., Wallon, G., & Ledin, A. (2008). Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European Molecular Biology Organization programmes. PLoS One, 3(10), e3480.
8. Brouns, M. (2000). The gendered nature of assessment procedures in scientific research funding: The Dutch case. Higher Education in Europe, 25(2), 193-199.
9. Bowen, D.D., Perloff, R., & Jacoby, J. (1972). Improving manuscript evaluation procedures. American Psychologist, 27, 221-225.
10. Cheung, S.F., & Chan, D.K.S. (2004). Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. Journal of Applied Psychology, 89(5), 780-791.
11. Cheung, S.F., & Chan, D.K.S. (2008). Dependent correlations in meta-analysis-The case of heterogeneous dependence. Educational and Psychological Measurement, 68(5), 760-777.
12. Chinn, S. (2000). A simple method for converting an odds ratio to effects size for use in meta-analysis. Statistics in Medicine, 19, 3127-3131.
13. Chubin, D.E. (1994). Grants peer review in theory and practice. Evaluation Review, 18, 20-30.
14. Cicchetti, D.V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14, 119-135.
15. Cicchetti, D.V., & Eron, L.D. (1979). The reliability of manuscript reviewing for the Journal of Abnormal Psychology. Paper presented at the 1979 Proceedings of the Social Statistics Section, Washington, DC.
16. Cohn, L.D., & Becker, B.J. (2003). How meta-analysis increases statistical power. Psychological Methods, 8, 243-253.
17. Cole, J.R. (1979). Fair science: Women in the scientific community. New York: Free Press.
18. Cooper, H., & Hedges, L. V. (Eds.) (1994). The handbook of research synthesis. New York: Russell Sage.
19. Cooper, H.M. (1998). Integrating research: A guide for literature reviews (3rd ed.). Newbury Park, CA: Sage.
20. Crane, D. (1967). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. American Sociologist, 32, 195-201.
21. Crane, D. (1970). The academic market place revisited. American Journal of Sociology, 7, 953-964.
22. Demicheli, V., & Pietrantonj, C. (2007). Peer review for improving the quality of grant applications (Cochrane Methodology Review). Cochrane Database of Systematic Reviews, Issue 2. Art. No.: MR000003.
23. Egger, M., Ebrahim, S., & Smith, G.D. (2002). Where now for meta-analysis? International Journal of Epidemiology, 31, 1-5.
24. Enders, C.K. (2005). Analyzing structural equation models with missing data. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (pp. 315-344). Greenwich, CT: Information Age.
25. Erez, A., Bloom, M.C., & Wells, M.T. (1996). Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization. Personnel Psychology, 49, 275-306.
26. Fox, M.F. (1991). Gender, environmental milieu, and productivity in science. In H. Zuckerman, J. R. Cole, & J. T. Bruer (Eds.), The outer circle: Women in the scientific community (pp. 188-204). New York: W. W. Norton.
27. Glass, G.V. (2000, January). Meta-analysis at 25. Retrieved April 22, 2009, from Arizona State University, College of Education Web site: http://glass.ed.asu.edu/gene/papers/meta25.html.
28. Gilbert, J.R., Williams, E.S., & Lundberg, G.D. (1994). Is there gender bias in JAMA's peer review process? JAMA, 272, 139-142.
29. Goldbeck-Wood, S. (1999). Evidence on peer review-Scientific quality control or smokescreen? BMJ, 7175, 44-45.
30. Goldstein, H. (2003). Multilevel statistical models (3rd ed.). London: Hodder Arnold.
31. Goldstein, H., Yang, M., Omar, R., Turner, R., & Thompson, S. (2000). Meta-analysis using multilevel models with an application to the study of class size effects. Journal of the Royal Statistical Society: Series C (Applied Statistics), 49, 399-412.
32. Gordon, M.D. (1980). The role of referees in scientific communication. In J. Hartley (Ed.), The psychology of written communication (pp. 263-275). London: Kogan Page.
33. Hedges, L.V., & Olkin, I. (1985). Statistical methods for the meta-analysis. New York: Academic Press.
34. Hedges, L.V., & Vevea, J.L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
35. Hyde, J.S. (2005). The gender similarities hypothesis. American Psychologist, 60, 581-592.
36. Hox, J.J. (2002). Applied multilevel analysis. Amsterdam, the Netherlands: TT Publishers.
37. Hox, J.J. (2008). Meta-analysis: Advanced multilevel approaches. Available athttp://www.education.ox.ac.uk/research/resgroup/self/training.php.
38. Hox, J.J., & de Leeuw, E.D. (2003). Multilevel models for meta-analysis. In S. P. Reise & N. Duan (Eds.), Multilevel modelling: Methodological advances, issues, and applications (pp. 90-111). Mahwah, NJ: Lawrence Erlbaum.
39. Jayasinghe, U.W. (2004). Peer review in the assessment and funding of research by Australian Research Council. Unpublished doctoral dissertation, University of Western Sydney, Australia.
40. Jayasinghe, U.W., Marsh, H.W., & Bond, N. (2001). Peer review in the funding of research in higher education: The Australian experience. Educational Evaluation and Policy Analysis, 23(4), 343-364.
41. Jayasinghe, U.W., Marsh, H.W., & Bond, N. (2003). A multilevel cross-classified modeling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society: Series A, 166(3), 279-300.
42. Jayasinghe, U.W., Marsh, H.W., & Bond, N. (2006). A new reader trial approach to peer review in funding research grants: An Australian experiment. Scientometrics, 69, 591-606.
43. Kalaian, H.A., & Raudenbush, S.W. (1996). A multivariate mixed-effects linear model for meta-analysis. Psychological Methods, 1, 227-235.
44. Kalaian, S.A., & Kasim, R.M. (2008). Applications of multilevel models for meta-analysis. In A. A. O'Connell & D. B. McCoach (Eds.), Multilevel analysis of educational data. Greenwich, CT: Information Age.
45. Lipsey, M.W., & Wilson, D.B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
46. Little, R.J.A., & Rubin, D.B. (1987). Statistical analysis with missing data. New York: John Wiley.
47. Long, J.S., Allison, P.D., & McGinnes, R. (1979). Entrance into the academic career. American Sociological Review, 44, 816-830.
48. Mahoney, M.J. (1978). Publish and perish. Human Behaviour, 7, 38-41.
49. Mahoney, M.J. (1985). Open exchange and epistemic progress. American Psychologist, 40(1), 29-39.
50. Mahoney, M.J. (1987). Scientific publication and knowledge politics. Journal of Social Behaviour and Personality, 2(2), 165-176.
51. Manwell, C., & Baker, M.A. (1982). Reform peer review: The Peters and Ceci study in the context of other current studies of scientific evaluation. Behavioral and Brain Sciences, 5, 221-225.
52. Marsh, H.W.l & Ball, S. (1981). Interjudgmental reliability of review for the Journal of Educational Psychology. Journal of Educational Psychology, 73, 872-880.
53. Marsh, H.W., & Ball, S. (1989). The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability. Journal of Experimental Education, 57, 151-169.
54. Marsh, H.W., & Ball, S. (1991). Reflections on the peer review process. Behavioral and Brain Sciences, 14, 157-158.
55. Marsh, H.W., & Hau, K.-T. (2007). Applications of latent-variable models in educational psychology: The need for methodological-substantive synergies. Contemporary Educational Psychology, 32, 151-171.
56. Marsh, H.W., Jayasinghe, U.W., & Bond, N. (2007). The peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid. Australian Psychologist, 42, 33-38.
57. Marsh, H.W., Jayasinghe, U.W., & Bond, N.W. (2008a). Improving the peer-review process for grant applications: Reliability, validity and generalization. American Psychologist, 63, 160-168.
58. Marsh, H.W., Jayasinghe, U.W., & Bond, N.W. (2008b). Peer reviews of grant applications: Effects of applicant and assessor gender. Oxford, UK: SELF Research Centre, University of Oxford.
59. Marsh, H.W., Martin, A.J., & Cheng, J.H.S. (2008). A multilevel perspective on gender in classroom motivation and climate: Potential benefits of male teachers for boys? Journal of Educational Psychology, 100, 78-95.
60. Marsh, H.W., O'Mara, A.J. & Malmberg, L. (2008). Meta-analysis. Available athttp://www.education.ox.ac.uk/research/resgroup/self/training.php.
61. Marsh, H.W., Wen, Z.L., Hau, K.T., Little, T.D., Bovaird, J.A., & Widaman, K.F. (2007). Unconstrained structural equation models of latent interactions: Contrasting residual-and mean-centered approaches. Structural Equation Modeling, 14, 570-580.
62. Merton, R.K. (1968). The Matthew effect in science. Science, 159, 56-63.
63. Merton, R.K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.
64. National Science Foundation. (2003). Gender differences in the careers of academic scientists and engineers: A literature review (special report). Arlington, VA: Author.
65. O'Mara, A.J., & Marsh, H.W. (2008). Incorporating within-study correlations in multivariate meta-analysis: Multilevel versus traditional models. Oxford, UK: SELF Research Centre, University of Oxford.
66. O'Mara, A.J., Marsh H.W., Craven, R.G., & Debus, R. (2006). Do self-concept interventions make a difference? A synergistic blend of construct validation and meta-analysis. Educational Psychologist, 41, 181-206.
67. Oxman A.D., Cook, D.J., & Guyatt, G.H. (1994). Users' guides to the medical literature. VI. How to use an overview. JAMA, 272(17), 1367-1371.
68. Peters, D.P., & Ceci, S.J. (1982). Peer-review practices of psychological journals- The fate of accepted, published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187-195.
69. Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2004). GLLAMM manual. Berkeley: University of California.
70. Raudenbush, S.W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.
71. Raudenbush, S.W. (1994). Random effects model. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 301-321). New York: Russell Sage.
72. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Newbury Park, CA: Sage.
73. Sandström, U., & Hällsten, M. (2008). Persistent nepotism in peer-review. Scientometrics, 74(2), 175-189.
74. Sheridan, B. (1998). Strangers in a strange land: A literature review of women in science (Working Paper No. 17). Washington, DC: World Bank.
75. Snijders, T.A.B., & Bosker, R.J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage Ltd.
76. StataCorp. (2005). Stata statistical software: Release 9 [computer software]. College Station, TX: Author.
77. Tregenza, T. (2002). Gender bias in the refereeing process? Trends in Ecology and Evolution, 17(8), 349-350.
78. Van den Noortgate, W., & Onghena, P. (2003). Multilevel meta-analysis: A comparison with traditional meta-analytic procedures. Educational and Psychological Measurement, 63, 765-790.
79. Wennerås, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341-343.
80. Wennerås, C., & Wold, N. (2000). A chair of one's own. The upper reaches of academe remain stubbornly inaccessible to women. Nature, 408(6813), 647.
81. Wood, F.Q., & Wessely, S. (2003). Peer review of grant applications: A systematic review. In F. Godlee & T. Jefferson (Eds.), Peer review in health sciences (2nd ed., pp. 14-44). London: BMJ Books.
82. Zuckerman, H. (1970). Stratification in American science. Sociological Inquiry, 40, 235-257.
83. Zuckerman, H., & Merton, R.K. (1971). Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, 9, 66-100.