A comparison of the generalizability of scores produced by expert raters and automated scoring systems

Applied Measurement in Education

Volumn 12, Issue 3, 1999, Pages 281-299

  Clauser, Brian E ^a   Swanson, David B ^a   Clyman, Stephen G ^a  

^a NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0041526020     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1207/S15324818AME1203_4     Document Type: Article

References (23)

1. Baxter, G. P., Shavelson, R. J., Goldman, S. R., & Pine, J. (1992). Evaluation of procedure-based scoring for hands-on science assessment. Journal of Educational Measurement, 29, 1-17.
2. Bejar, I. I. (1991). A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology, 76, 522-532.
3. Bennett, R. E., & Sebrechts, M. M. (1996). The accuracy of expert-system diagnoses of mathematical problem solutions. Applied Measurement in Education, 9, 133-150.
4. Braun, H. I., Bennett, R. E., Frye, D., & Soloway, E. (1990). Scoring constructed responses using expert systems. Journal of Educational Measurement, 27, 93-108.
5. Brennan, R. L. (1996). Generalizability of performance assessments. In G. W. Phillips (Ed.), Technical issues in large-scale performance assessments (pp. 19-58). Washington, DC: National Center for Educational Statistics.
6. Clauser, B. E., Clyman, S. G., & Swanson, D. B. (1999). Components of rater error in a complex performance assessment. Journal of Educational Measurement, 36, 29-45.
7. Clauser, B. E., Margolis, M. J., Clyman, S. G., & Ross, L. P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal of Educational Measurement, 34, 141-161.
8. Clauser, B. E., Ross, L. P., Clyman, S. G., Rose, K. M., Margolis, M. J., Nungester, R. J., Piemme, T. E., Chang, L., El-Bayoumi, G., Malakoff, G. L., & Pincetl, P. S. (1997). Development of a scoring algorithm to replace expert rating for scoring a complex performance-based assessment. Applied Measurement in Education, 10, 345-358.
9. Clauser, B. E., Subhiyah, R. G., Nungester, R. J., Ripkey, D. R., Clyman, S. G., & McKinley, D. (1995). Scoring a performance-based assessment by modelling the judgments of experts. Journal of Educational Measurement, 32, 397-415.
10. Clauser, B. E., Swanson, D. B., & Clyman, S. G. (1996). The generalizability of scores from a performance assessment of physicians' patient management skills. Academic Medicine, 71(10 RIME Suppl.), S109-S111.
11. Clyman, S. G., Melnick, D. E., & Clauser, B. E. (1995). Computer-based case simulations. In E. L. Mancall & P. G. Bashook (Eds.), Assessing clinical reasoning: The oral examination and alternative methods (pp. 139-149). Evanston, IL: American Board of Medical Specialities.
12. Crick, J. E., & Brennan, R. L. (1983). Manual for GENOVA: A generalized analysis of variance system. Iowa City, Iowa: ACT Publications.
13. Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
14. Engelhard, G. (1994). Examining rater errors in the assessment of written composition with a many faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
15. Hardy, R. A. (1995). Examining the costs of performance assessment. Applied Measurement in Education, 8, 121-134.
16. Kaplan, R. M., & Bennett, R. E. (1994). Using a free-response scoring tool to automatically score the formulating-hypotheses item (RR 94-08). Princeton, NJ: Educational Testing Service.
17. Linn, R. L., & Burton, E. (1994). Performance-based assessment: Implications of task specificity. Educational Measurement: Issues and Practice, 13(1), 5-8, 15.
18. Longford, N. T. (1995). Models for uncertainty in educational testing. New York: Springer-Verlag.
19. Norcini, J. J., Diserens, D., Day, S. C., Cebul, R. C., Schwartz, J. S., Beck, L. H., Webster, G. D., Schnabel, T. G., & Elstein, A. S. (1990). The scoring and reproducibility of an essay test of clinical judgment. Academic Medicine, 65(9 RIME Suppl.), S41-S42.
20. Page, E. B., & Petersen, N. S. (1995). The computer moves into essay grading. Phi Delta Kappan, 76, 561-565.
21. Reckase, M. D. (1995). The reliability of ratings versus the reliability of scores. Educational Measurement: Issues and Practice, 14(4), 31.
22. Sebrechts, M. M., Bennett, R. E., & Rock, D. A. (1991). Agreement between expert-system and human raters on complex constructed-response quantitative items. Journal of Applied Psychology, 76, 856-862.
23. Wainer, H., & Thissen, D. (1993). Combining multiple-choice and constructed response test scores: Toward a Marxist theory of test construction. Applied Measurement in Education, 6, 103-118.