Latent Markov modeling applied to grant peer review

Journal of Informetrics

Volumn 2, Issue 3, 2008, Pages 217-228

  Bornmann, Lutz ^a   Mutz, Rüdiger ^a   Daniel, Hans Dieter ^a,b  

^a ETH ZURICH   (Switzerland)

^b UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Latent class analysis; Latent Markov model; Multi stage evaluation process; Peer review; Reliability

Indexed keywords

EID: 47149091087     PISSN: 17511577     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.joi.2008.05.003     Document Type: Article

References (38)

1. Agresti A. Analyzing repeated categorical response data. In: Agresti A. (Ed). Categorical data analysis. 2nd ed. (2002), Wiley, New York, NY, USA 455-490
2. Bornmann L., and Daniel H.-D. Committee peer review at an international research foundation: Predictive validity and fairness of selection decisions on post-graduate fellowship applications. Research Evaluation 14 1 (2005) 15-20
3. Bornmann L., and Daniel H.-D. Criteria used by a peer review committee for selection of research fellows-A boolean probit analysis. International Journal of Selection and Assessment 13 4 (2005) 296-303
4. Bornmann L., and Daniel H.-D. Selection of research fellowship recipients by committee peer review. Analysis of reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics 63 2 (2005) 297-320
5. Bornmann L., and Daniel H.-D. Potential sources of bias in research fellowship assessments. Effects of university prestige and field of study on approval and rejection of fellowship applications. Research Evaluation 15 3 (2006) 209-219
6. Bornmann L., and Daniel H.-D. Selecting scientific excellence through committee peer review-A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics 68 3 (2006) 427-440
7. Bornmann L., and Daniel H.-D. Convergent validation of peer review decisions using the h index: Extent of and reasons for type I and type II errors. Journal of Informetrics 1 3 (2007) 204-213
8. Bornmann L., and Daniel H.-D. Gatekeepers of science-Effects of external reviewers' attributes on the assessments of fellowship applications. Journal of Informetrics 1 1 (2007) 83-91
9. Bornmann L., Mutz R., and Daniel H.-D. Row-column (RC) association model applied to grant peer review. Scientometrics 73 2 (2007) 139-147
10. Camerer C.F., and Fehr E. When does "economic man" dominate social behavior?. Science 311 5757 (2006) 47-52
11. Cicchetti D.V. The reliability of peer-review for manuscript and grant submissions-A cross-disciplinary investigation. Behavioral and Brain Sciences 14 1 (1991) 119-134
12. Daniel, H.-D. (1993/2004). Guardians of science. Fairness and reliability of peer review (chapter: Reliability of manuscript refeering). Weinheim, Germany: Wiley-VCH. Published online 16 July 2004, Wiley Interscience. doi:10.1002/3527602208.
13. Fröhlich H. It all depends on the individuals. Research promotion-A balanced system of control. B.I.F. Futura 16 (2001) 69-77
14. Fröhlich H. In the hands of social researchers. B.I.F. Futura 19 (2004) 19-23
15. Hodgson C. Evaluation of cardiovascular grant-in-aid applications by peer review: Influence of internal and external reviewers and committees. Canadian Journal of Cardiology (1995) 864-868
16. Jayasinghe U.W., Marsh H.W., and Bond N. A multilevel cross-classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society Series a-Statistics in Society 166 (2003) 279-300
17. Klahr D. Insiders, outsiders, and efficiency in a National Science Foundation panel. American Psychologist 40 2 (1985) 148-154
18. Langeheine R. Manifest and latent Markov chain models for categorical panel data. Journal of Educational Statistics 13 4 (1988) 299-312
19. Langeheine R. Latent variables Markov models. In: von Eye A., and Clogg C.C. (Eds). Latent variables analysis (1994), Sage, Thousand Oaks, CA, USA 373-395
20. Langeheine R., Pannekoek J., and van de Pol F. Bootstrapping goodness-of-fit measures in categorical data analysis. Sociological Methods and Research 24 4 (1996) 492-516
21. Langeheine R., and van de Pol F. A unifying framework for Markov modeling in discrete space and discrete time. Sociological Methods and Research 18 4 (1990) 416-441
22. Langeheine R., and van de Pol F. Applied latent class analysis. In: Hagenaars J.A., and McCutcheon A.L. (Eds). Applied latent class analysis (2002), University Press, Cambridge, UK 304-341
23. Marsh H.W., Jayasinghe U.W., and Bond N.W. Improving the peer-review process for grant applications - reliability, validity, bias, and generalizability. American Psychologist 63 3 (2008) 160-168
24. Marshall E. NIH tunes up peer-review. Science 263 5151 (1994) 1212-1213
25. McCutcheon A. L. (1987). Latent class analysis (chapter: Latent class analysis). Newbury Park, CA, USA: Sage.
26. Muthén L. K., & Muthén B. O. (1998-2006). Mplus user's guide (4th ed., chapter: Mixture modelling with longitudinal data). Los Angeles, CA, USA: Muthén & Muthén.
27. Novick M.R. The axioms and principal results of classical test theory. Journal of Mathematical Psychology 3 (1966) 1-18
28. Office of Management and Budget. Revised information quality bulletin for peer review (2004), Office of Management and Budget, Washington, DC, USA
29. Peterson R.A. A meta-analysis of Cronbach's coefficient alpha. The Journal of Consumer Research 21 2 (1994) 381-391
30. Poulsen, C. S. (1982). Latent structure analysis with choice modeling applications (chapter: Models with latent changes). Aarhus, Denmark: The Aarhus School of Business Administration and Economics.
31. Poulsen C.S. Mixed Markov and latent Markov modelling applied to brand choice behaviour. International Journal of Research in Marketing 7 (1990) 5-19
32. Singer B., and Spilerman S. The representation of social processes by Markov models. The American Journal of Sociology 82 1 (1976/77) 1-54
33. United States General Accounting Office. Peer review practices at federal science agencies vary (1999), United States General Accounting Office, Washington, DC, USA
34. van de Pol, F., Langeheine, R., & de Jong, W. (2000). PANMARK 3: User's manual: PANel analysis using MARKov chains; a latent class analysis program (chapter: Introduction). Voorburg: Netherlands Central Bureau of Statistics.
35. Vener K.J., Feuer E.J., and Gorelic L. A statistical model validating triage for the peer-review process-Keeping the competitive applications in the review pipeline. FASEB Journal 7 14 (1993) 1312-1319
36. von Eye, A., & Mun, E. Y. (2005). Analyzing rater agreement. Manifest variable methods (chapter: Coefficients of rater agreement). Mahwah, NJ, USA: Lawrence Erlbaum Associates.
37. Wiggins, L. M. (1973). Panel analysis-Latent probability models for attitude and behavior processes (chapter: Models involving both latent change and change in latent probabilities). New York, NY, USA: Elsevier.
38. Wood F.Q., and Wesseley S. Peer review of grant applications: A systematic review. In: Godlee F., and Jefferson T. (Eds). Peer review in health sciences. 2nd ed. (2003), BMJ Books, London, UK 14-44