A Bayesian approach to ranking and rater evaluation: An application to grant reviews

Journal of Educational and Behavioral Statistics

Volumn 35, Issue 2, 2010, Pages 194-214

  Cao, Jing ^a   Stokes, S Lynne ^a   Zhang, Song ^b  

^a SOUTHERN METHODIST UNIVERSITY   (United States)

^b UNIVERSITY OF TEXAS SOUTHWESTERN MEDICAL CENTER   (United States)

Author keywords

Bayesian hierarchical model; Multirater ordinal data; Number of raters; Ranking estimator; Rater performance

Indexed keywords

EID: 77952527478     PISSN: 10769986     EISSN: None     Source Type: Journal    
DOI: 10.3102/1076998609353116     Document Type: Review

References (28)

1. Agresti, A. (1988). A model for agreement between ratings on an ordinal scale. Biometrics, 44, 539-548.
2. Albert, J.A., & Chib, S. (1993). Bayesian analysis of binary of polychotomous response data. Journal of the American Statistical Association, 88, 669-679.
3. Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561-573.
4. Basu, S., Basu, A., & Raychaudhuri, A. (1999). Measuring agreement between two raters for ordinal response: A model-based approach. Journal of the Royal Statistical Society. Series D, 48, 339-348.
5. Berger, J.O. (1985). Statistical decision theory and Bayesian analysis (2nd ed.). New York: Springer.
6. Bonett, D.G. (2002). Sample size requirement for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics, 27, 335-340.
7. Graham, P., & Jackson, R. (1993). The analysis of ordinal agreement data: Beyond weighted kappa. Journal of Clinical Epidemiology, 46, 1055-1062.
8. Ishwaran, H. (2000). Univariate and multirater ordinal cumulative link regression with covariate specific cutpoints. Canadian Journal of Statistics, 28, 715-730.
9. Ji, Y., Lu, Y., & Mills, G.B. (2008). Bayesian models based on test statistics for multiple hypothesis testing problems. Bioinformatics, 24, 943-949.
10. Johnson, V.E. (1996). On Bayesian analysis of multirater ordinal data: An application to automated essay grading. Journal of the American Statistical Association, 91, 42-51.
11. Johnson, V.E. (2007). Bayesian model assessment using pivotal quantities. Bayesian Analysis, 2, 1-15.
12. Johnson, V.E. (2008). Statistical analysis of the National Institute of Health peer review system. Proceedings of the National Academy of Sciences, 105, 11076-11080.
13. Johnson, V.E., & Albert, J.A. (1999). Ordinal data modeling. New York: Springer.
14. Kruskal, W.H. (1958). Ordinal measures of association. Journal of the American Statistical Association, 53, 814-861.
15. Laird, N.M., & Louis, T.A. (1989). Empirical Bayes ranking methods. Journal of Educational Statistics, 14, 29-46.
16. Lin, R., Louis, T.A., Paddock, S.M., & Ridgeway, G. (2006). Loss function based ranking in two-stage, hierarchical models. Bayesian Analysis, 1, 915-946.
17. Master, G.N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
18. Mellenberg, G.J. (1994). Generalized linear item response theory. Psychological Bulletin, 115, 300-307.
19. Perkins, S.M., & Becker, M.P. (2002). Assessing rater agreement using marginal association models. Statistics in Medicine, 21, 1743-1760.
20. Saito, Y., Sozu, T., Hamada, C., & Yoshimura, I. (2006). Effective number of subjects and number of raters for inter-rater reliability studies. Statistics in Medicine, 25, 1547-1560.
21. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, No. 17.
22. Samejima, F. (1972). A general model for free response data. Psychometrika, 38, 221-233.
23. Shen, W., & Louis, T.A. (1998). Triple-goal estimates in two-stage hierarchical models. Journal of the Royal Statistical Society. Series B, 60, 455-471.
24. Spiegelhalter, D.J., Abrams, K.R., & Myles, J.P. (2004). Bayesian approaches to clinical trials and health-care evaluation. Chichester, UK: Wiley.
25. Thissen, D., & Steinberg, L. (1986). A taxonomy of item response models. Psychometrika, 51, 567-577.
26. Uebersax, J., & Grove, W.M. (1993). A latent trait finite-mixture model for the analysis of rating agreement. Biometrics, 49, 823-835.
27. van der Linden, W. J. & Hambleton, R. K. (Eds.). (1997). Handbook of modern item response theory. New York: Springer-Verlag.
28. Young, H.P. (1988). Condorcet's theory of voting. American Political Science Review, 82, 1231-1244.