A multilevel cross-classified modelling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings

Journal of the Royal Statistical Society. Series A: Statistics in Society

Volumn 166, Issue 3, 2003, Pages 279-300

  Jayasinghe, Upali W ^a   Marsh, Herbert W ^a   Bond, Nigel ^a  

^a UNIVERSITY OF WESTERN SYDNEY   (Australia)

Author keywords

Australian research council; Cross classified models; Grant proposal funding; Interrater reliability; Multilevel modelling; Peer review process

Indexed keywords

EID: 0141888420     PISSN: 09641998     EISSN: None     Source Type: Journal    
DOI: 10.1111/1467-985X.00278     Document Type: Review

References (34)

1. Allison, P. D. and Stewart, J. A. (1974) Productivity differences among scientists: evidence for accumulative advantage. Am. Sociol. Rev., 39, 596-606.
2. Australian Research Council (1996) Large research grants guidelines for 1996. In ARC Members' Handbook 1996. Canberra: Australian Research Council.
3. Bazeley, P. (1998) Peer review and Panel decisions in the assessment of Australian Research Council Project grant applicants: what counts in a highly competitive context? Higher Educ., 35, 435-452.
4. Bazeley, P., Kemp, L., Stevens, K., Asmar, C., Grbich, C., Marsh, H. W. and Bhatal, R. (1996) Waiting in the Wings: a Study of Early Career Academic Researchers in Australia. Canberra: National Board of Employment, Education and Training.
5. Bond, N. and Hesketh, B. (1998) A report on the "Reader Trial" conducted by Panel A7a-Psychology and Education. University of Western Sydney, Sydney.
6. Cicchetti, D. V. (1991) The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behav. Brain Sci., 14, 119-135.
7. Cole, S., Rubin, L. and Cole, J. R. (1977) Peer review and the support of science. Scient. Am., 237, 34-41.
8. Daniel, H. D. (1993) Guardians of Science: Fairness and Reliability of Peer Review (transl. by E. R. William). Weinnheim: VCH Weinnheim.
9. Emery, J. A., Meyers, H. W. and Hunter, D. E. (1992) NIH FIRST awards: testing background factors for funding against peer review. J. Soc. Res. Admin., 24, part 2, 7-15.
10. Garfield, E. (1986) Refereeing and peer review: Part 1, opinion and conjecture on the effectiveness of refereeing. Curr. Contents, 31, 3-11.
11. Gilbert, J. R., Williams, E. S. and Lundberg, G. D. (1994) Is there gender bias in JAMA's peer review process? J. Am. Med. Ass., 272, 139-142.
12. Goldstein, H. (1995) Multilevel Statistical Model, 2nd edn. London: Arnold.
13. Goldstein, H. and Healy, M. J. R. (1995) The graphical presentation of a collection of means. J. R. Statist. Soc. A, 158, 175-177.
14. Goldstein, H., Rasbash, J., Plewis, I., Draper, D., Browne, W., Yang, M., Woodhouse, G. and Healy, M. (1998) A User's Guide to MLwiN. London: Institute of Education.
15. Hammel, E. (1980) Report of the Task Force on Faculty Renewal. Population Research Center, University of California at Berkeley, Berkeley.
16. Lindsey, D. (1976) Distinction achievement and editorial board membership. Am. Psychol., 31, 799-804.
17. Lindsey, D. (1977) Participation and influence in publication review proceedings: a reply. Am. Psychol., 32, 579-585.
18. Lindsey, D. (1978) The Scientific Publication System in Social Science. San Francisco: Jossey-Bass.
19. Mahoney, M. J. (1985) Open exchange and epistemic progress. Am. Psychol., 40, 29-39.
20. Marsh, H. W. and Ball, S. (1981) The interjudgmental reliability of reviews for the Journal of Educational Psychology. J. Educ. Psychol., 73, 872-880.
21. Marsh, H. W. and Ball, S. (1989) The peer review process used to evaluate manuscripts submitted to academic journals: interjudgemental reliability. J. Exptl Educ., 57, 151-169.
22. Marsh, H. W. and Ball, S. (1991) Reflections on the peer review process. Behav. Brain Sci., 14, 157-158.
23. Marsh, H. W. and Bazeley, P. (1999) Multiple evaluations of grant proposals by independent assessors: confirmatory factor analysis, evaluations of reliability, validity and structure. Multiv. Behav. Res., 34, 1-30.
24. Naylor, C. D. (1994) Reviewing applied research grant proposals: can we learn from medical journals? Can. Med. Ass. J., 150, 1207-1209.
25. Neidhardt, F. (1988) Selbststeuerung in der Forschungsförderung: das Gutachterwesen der DFG Opladen. Westdeutscher Verlag.
26. Osmond, D. H. (1983) Malice's Wonderland: research funding and peer review. J. Neurbiol., 14, 95-112.
27. Perlman, D. (1982) Reviewer "bias": do Peters and Ceci protest too much? Behav. Brain Sci., 5, 231-232.
28. Rasbash, J., Browne, W., Goldstein, H., Yang, M., Plewis, I., Healy, M., Woodhouse, G., Draper, D., Langford, I. and Lewis, T. (1999) A User's Guide to MLwiN - Version 2.0. London: Institute of Education.
29. Rasbash, J. and Goldstein, H. (1994) Efficient analysis of mixed hierarchical and cross-classified random structures using a multilevel model. J. Educ. Behav. Statist., 19, 337-350.
30. Shout, P. E. and Fleiss, J. L. (1979) Intraclass correlations: uses in assessing rater reliability. Psychol. Bull., 86, 420-428.
31. Snijders, T. A. B. and Bosker, R. J. (1999) Multilevel Analysis: an Introduction to Basic and Advanced Multilevel Modeling. London: Sage.
32. Winer, B. J. (1971) Statistical Principles in Experimental Design. New York: McGraw-Hill.
33. Wood, F. Q. (1997) The Peer Review Process. Canberra: National Board of Employment, Education and Training.
34. Zuckerman, H. and Merton, R. K. (1971) Patterns of evaluation in science: institutionalisation, structure and functions of the referee system. Minerva, 9, 66-100.