A new reader trial approach to peer review in funding research grants: An Australian experiment

Scientometrics

Volumn 69, Issue 3, 2006, Pages 591-606

  Jayasinghe, Upali W ^a   Marsh, Herbert W ^a,b   Bond, Nigel ^a  

^a UNIVERSITY OF WESTERN SYDNEY   (Australia)

^b UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33846470568     PISSN: 01389130     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11192-006-0171-4     Document Type: Article

References (44)

1. AUSTRALIAN RESEARCH COUNCIL (ARC). (1996), ARC Members Handbook. Australian Research Council.
2. BAZELEY, P. (1998), Peer review and panel decisions in the assessment of Australian Research Council Project grant applicants: What counts in a highly competitive context? Higher Education, 35 : 435-452.
3. BORNMANN, L., DANIEL, H. D. (2005), Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustee decisions. Scientometrics, 63 : 297-320.
4. BOWEN, D. D., PERLOFF, R., JACOBY, J. (1972), Improving manuscript evaluation procedures. American Psychologist, 27 : 221-225.
5. BROWNE, W. J. (2002), MCMC Estimation in MLwiN by Centre for Multilevel Modelling: Institute of Education, University of London.
6. BROWNE, W. J., GOLDSTEIN, H., RASBASH, J. (2001), MCMC membership multiple classification (MMMC) models. Journal of Statistical Modelling, 1 : 103-124.
7. CALLAHAM, M. L., BAXT, W. G., WAECKERLE, J. F., WEARS, R. L. (1998), Reliability of editors' subjective quality ratings of peer reviews of manuscripts. JAMA: Journal of the American Medical Association, 280 : 229-231.
8. CHUBIN, D. E. (1994), Grants peer review in theory and practice. Evaluation Review, 18 : 20-30.
9. CICCHETTI, D. V. (1980), Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist, 35 : 300-303.
10. CICCHETTI, D. V. (1991), The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioural and Brain Sciences, 14 : 119-135.
11. DANIEL, H. D. (1993), Guardians of Science: Fairness and Reliability of Peer Review. Wiley: New York.
12. FRESHWATER, D. (Ed.) (2005), Editorial: Re-viewing (A year in the life). Journal of Psychiatric & Mental Health Nursing, 12 : 385-386.
13. GARFIELD, E. (1986), Refereeing and peer review, Part 1. Opinion and conjecture on the effectiveness of refereeing. Current Contents, 31 : 3-11.
14. GOLDBECK-WOOD, S. (1999), Evidence on peer review - scientific quality control or smokescreen? BMJ: British Medical Journal, 7175 : 44-45.
15. GOLDSTEIN, H. (1995), Multilevel Statistical Models (2nd ed.). London: Edward Arnold.
16. HODOSON, C. (1995), Evaluation of cardiovascular grant-in-aid applications by peer review - Influence of internal and external reviewers and committees. Canadian Journal of Cardiology, 11 (10) : 864-868.
17. HOJAT, M., GONNELLA, J. S., CAELLEIGH, A. S. (2003), Impartial judgment by the "gatekeepers" of science: Fallibility and accountability in the peer review process. Advances in Health Sciences Education, 8 : 75-96.
18. JAYASINOHE, U. W. (2004), Peer Review in the Assessment and Funding of Research by Australian Research Council. Doctoral dissertation submitted to the University of Western Sydney, Australia.
19. JAYASINOHE, U. W., MARSH, H. W., BOND, N. (2001), Peer review in the funding of research in higher education: The Australian experience. Educational Evaluation and Policy Analysis, 23 (4) : 343-364.
20. JAYASINGHE, U. W., MARSH, H. W., BOND, N. (2003), A multilevel cross-classified modeling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society (A), 166 (3) : 279-300.
21. KASSIRER, J. P., CAMPION, E. W. (1994), Peer review: Crude and understudied, but indispensable. JAMA: Journal of the American Medical Association, 272 : 96-97.
22. KLAHR, D. (1985), Insiders, outsiders and efficiency in a National Science Foundation panel. American Psychologist, 40 (2) : 148-154.
23. LAMING, D. (1990), The reliability of a certain university examination compared with the precision of absolute judgments. Quarterly Journal of Experimental Psychology, 42A : 239-254.
24. LEOPOLD, A. C. (1978), The act of creation: Creative processes in science. Bio Science, 28 : 436-440.
25. LUNDBERG, G. D. (2002), The publishing dilemma of the American Psychological Association. American Psychologist, 57 : 211-212.
26. MAHONEY, M. J. (1985), Open exchange and epistemic progress. American Psychologist, 40 : 29-39.
27. MANWELL, C., BAKER, M. A. (1982), Reform peer review: The Peters and Ceci study in the context of other current studies of scientific evaluation. Behavioural and Brain Sciences, 5 : 221-225.
28. MARSH, H. W., BALL, S. (1981), Interjudgmental reliability of review for the Journal of Educational Psychology. Journal of Educational Psychology, 73 : 872-880.
29. MARSH, H. W., BALL, S. (1989), The peer review process used to evaluate manuscripts submitted to academic journals: Interjudgmental reliability. Journal of Experimental Education, 57 : 151-169.
30. MARSH, H. W., BALL, S. (1991), Reflections on the peer review process. Behavioural and Brain Sciences, 14 : 157-158.
31. MARSH, H. W., BAZELEY, P. (1999), Multiple evaluations of grant proposals by independent assessors: Confirmatory factor analysis, evaluations of reliability, validity and structure. Multivariate Behavioural Research, 34 (1) : 1-30.
32. MARTIN, B. R. (1996), The use of multiple indicators in the assessment of basic research, Scientometrics, 36 : 343-362.
33. OVER, R. (1996), Perceptions of the Australian Research Council large grants scheme: Differences between successful and unsuccessful applicants. Australian Educational Researcher, 23 : 17-36.
34. PERLMAN, D. (1982), Reviewer "bias": Do Peters and Ceci protest too much? Behavioural and Brain Sciences, 5 : 231-232.
35. PETERS, D. P., CECI, S. J. (1982), Peer-review practices of psychological journals: The fate of published articles, submitted again. The Behavioural and Brain Sciences, 5 : 187-195.
36. RASBASH, J., STEELE, F., BROWNE, W., PROSSER, B. (2004), A User's Guide to MLwiN-Version 2.0. Institute of Education, University of London.
37. ROBERTS, L. W., COVERDALE, J., EDENHARDER, K., LOUIE, A. (2004), How to review a manuscript: A "down-to- earth" approach. Academic Psychiatry, 28 : 81-87.
38. ROSS, P. F. (1980), The Sciences' Self-Management: Manuscript Refereeing, Peer Review, and Goals in Science. Lincoln, Mass: The Ross Company.
39. SHROUT, P. E., FLEISS, J. L. (1979), Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin., 86 (3) : 420-428.
40. SNIJDERS, T. A. B., BOSKER, R. J. (1999), Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. London: Sage Publications.
41. VAN RAAN, A. F. J. (1996), Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises, Scientometrics, 36 : 397-420.
42. WEIL, J. (2004), Peer review: An essential step in the publishing process. Journal of Genetic Counseling, 13 : 183-187.
43. WEINGART, P. (2005), Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics, 62 : 117-131.
44. WOOD, F. Q. (1997), The Peer Review Process (Report No. 54). Canberra: National Board of Employment, Education and Training.