The generalizability of scores for a performance assessment scored with a computer-automated scoring system

Journal of Educational Measurement

Volumn 37, Issue 3, 2000, Pages 245-261

  Clauser, Brian E ^a   Harik, Polina ^a   Clyman, Stephen G ^a  

^a NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0034257275     PISSN: 00220655     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2000.tb01085.x     Document Type: Article

References (35)

1. Baker, E. L. (1992). The role of domain specification in improving the technical quality of performance assessment. Los Angeles, CA: UCLA, Center for Research on Ealuation, Standards, and Student Testing.
2. Behar, I. I. (1991). A methodology for scoring open-ended architectural desing problems. Journal of Applied Psychology, 76, 522-532.
3. Behar, I. I. & Baun, H. I. (1999). Achitectural simulations: From research to implementation. (Research Memorandum RM-990-2). Princeton, NJ: Educational Testing Service.
4. Bennett, R. E., & Sebrechts, M. M. (1994). The accuracy of automatic qualitative analyses of constructed-response solutions to algebra word problems (RR-94-04). Princeton, NJ: Educational Testing Service.
5. Braun, H. I., Bennett, R. E., Frye, D., & Soloway, E. (1990). Scoring constructed responses using expert systems. Journal of Educational Measurement, 27, 93-108.
6. Brennan, R. L. (1992). Elements of generalizability theory. Iowa City: ACT Publications.
7. Brennan, R. L. (1996). Generalizability of performance assessments. In G. W. Phillips (Ed.), Technical issues in large-scale performance assessments. Washington, DC: National Center for Educational Statistics.
8. Brennan, R. L. (1999). Manual for mGENOVA. Iowa Testing Programs Occasional Papers Number 47.
9. Burstein, J., Kukich, K., Wolff, S., & Lu, C. (1998, April). Computer analysis of essay content for automated score prediction. Paper presented at the meeting of the National Council on Measurement in Education, San Diego.
10. Case, S. M., Swanson, B. B., Ripkey, D. R. (1994). Comparison of items in five-option and extended-matching formats for the assessment of diagnostic skills. Academic Medicine (RIME Supplement), 69, S1-S3.
11. Clauser, B. E., & Clyman, S. G. (April, 1999). A strategy for increasing the generalizability of scores for performance assessments scored with automated scoring algorithms. Paper presented at the meeting of The American Educational Research Association, Montreal.
12. Clauser, B. E., Clyman, S. G., & Swanson, D. B. (1999). Components of rater error in a complex performance assessment. Journal of Educational Measurement, 36, 29-45.
13. Clauser, B. E., Margolis, M. J., Clyman, S. G., & Ross, L. P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal of Educational Measurement, 34, 141-161.
14. Clauser, B. E., Ross, L. P., Fan, V. Y., & Clyman, S. G. (1998). A comparison of two approaches for modeling expert judgment in scoring a performance assessment of physicians' patient-management skills. 73 (RIME Supplement), S117-S119.
15. Clauser, B. E., Subhiyah R., Nungester, R. J., Ripkey, D. R., Clyman, S. G., & McKinley, D. (1995). Scoring a performance-based assessment by modeling the judgements of experts. Journal of Educational Measurement, 32, 397-415.
16. Clauser, B. E., Swanson, D. B., & Clyman, S. G. (1996). The generalizability of scores from a performance assessment of physicians' patient management skills. Academic Medicine, 71 (RIME Supplement), S109-S111.
17. Clauser, B. E., Swanson, D. B., & Clyman, S. G. (1999). A comparison of the generalizability of scores produced by expert raters and automated scoring systems. Applied Measurement in Education, 12, 281-299.
18. Clyman, S. G., Melnick, D. E., & Clauser, B. E. (1999). Computer-based case simulations from medicine: Assessing skills in patient management. In A. Tekian, C. H. McGuire, & W. C. McGahie (Eds.), Innovative simulations for assessing professional competence (pp. 29-41). Chicago: University of Illinois, Department of Medical Education.
19. Crick, J. E., & Brennan, R. L. (1983). Manual for GENOVA: A generalized analysis of variance system. Iowa City, Iowa: ACT Publications.
20. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley & Sons, Inc.
21. Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E.H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373-399.
22. Jarjoura, D., & Brennan, R. L. (1982). A variance components model for measurement procedures associated with a table of specification. Applied Psychological Measurement, 6, 161-171.
23. Jarjoura, D., & Brennan, R. L. (1983). Multivariate generalizability analysis for tests developed from tables of specification. In L. J. Fyans (Ed.), Generalizability theory: Inferences and practical applications (pp. 85-101). San Francisco: Jossey-Bass Inc.
24. Kaplan, R. M., & Bennett, R. E. (1994). Using a free-response scoring tool to automatically score the formulating-hypotheses item (RR 94-08). Princeton, NJ: Educational Testing Service.
25. Keller, L. A., Clauser, B., Swanson, D., Harik, P., & Clyman, S. (2000, April). A multivariate generalizability analysis to examine the effect of content stratification on the reliability of a performance assessment. Paper presented at the meeting of the National Council on Measurement in Education, New Orleans, LA.
26. Linn, R. L., & Burton, E. (1994). Performance-based assessment: Implications of task specificity. Educational Measurement: Issues and Practice, 13, 5-8, 15.
27. Page, E. B. (1996). Grading essays by computer: Progress report. Proceedings of the 1966 invitational conference on testing. Princeton NJ: Educational Testing Service.
28. Page, E. B., & Petersen, N. S. (1995). The computer moves into essay grading. Phi Delta Kappan, 76, 561-565.
29. Ross, L. P., Clauser, B. E., Fan, V. Y., & Clyman, S. G. (1998, April). An examination of alternative models for automated scoring of a computer-delivered performance assessment. Paper presented at the meeting of The American Educational Research Association, San Diego.
30. Sebrechts, M. M., Bennett, R. E., & Rock, D. A. (1991). Agreement expert-system and human raters on complex constructed-response quantitative items. Journal of Applied Psychology, 76, 856-862.
31. Shavelson, R. J., Baxter, G. P., Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215-232.
32. Swanson, D. B., Clauser, B. E., Case, S. M. (1999). Clinical skills assessment with standardized patients in high-stakes tests: A framework for thinking about score precision, equating, and security. Advances in Health Sciences Education, 4, 67-106.
33. van der Vleuten, C.P.M., & Swanson, D. B. (1990). Assessment of clinical skills with standardized patients: The state of the art. Teaching and Learning in Medicine, 2, 58-76.
34. Webb, N. M., Shavelson, R., & Maddahian, E. (1983). Multivariate generalizability theory. In L. J. Fyans (Ed.), Generalizability theory: Inferences and practical applications (pp. 67-81). San Francisco: Jossey-Bass Inc.
35. Williamson, D., Bejar, I., & Hone, A. (1999). "Mental model" comparison of automated and human scoring. Journal of Educational Measurement, 36, 158-184.