Using multivariate generalizability theory to assess the effect of content stratification on the reliability of a performance assessment

Advances in Health Sciences Education

Volumn 15, Issue 5, 2010, Pages 717-733

  Keller, Lisa A ^a   Clauser, Brian E ^a   Swanson, David B ^a  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

Author keywords

Generalizability theory; Performance assessment; Reliability

Indexed keywords

ANALYSIS OF VARIANCE; ARTICLE; CLINICAL COMPETENCE; COMPUTER SIMULATION; HUMAN; MULTIVARIATE ANALYSIS; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICS; UNITED STATES; VALIDATION STUDY;

ANALYSIS OF VARIANCE; CLINICAL COMPETENCE; COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; HUMANS; MULTIVARIATE ANALYSIS; REPRODUCIBILITY OF RESULTS; STATISTICS AS TOPIC; UNITED STATES;

EID: 78649904815     PISSN: 13824996     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10459-010-9233-8     Document Type: Article

References (25)

1. Brennan, R. L. (1983). GENOVA [Computer Software]. Iowa City, IA: University of Iowa.
2. Brennan, R. L. (1999a). Manual for urGENOVA. Iowa testing programs occasional papers number 46.
3. Brennan, R. L. (1999b). Manual for mGENOVA. Iowa testing programs occasional papers number 47.
4. Brennan, R. L. (2001). Generalizability theory. New York: Springer.
5. Clauser, B. E., Harik, P., & Margolis, M. J. (2006). A multivariate generalizability analysis of data from a performance assessment of physicians' clinical skills. Journal of Educational Measurement, 43, 173-191.
6. Clauser, B. E., Harik, P., Margolis, M. J., Mee, J., Swygert, K., & Rebbecchi, T. (2008). The generalizability of documentation scores from the USMLE step 2 clinical skills examination. Academic Medicine (RIME Supplement).
7. Clauser, B. E., Swanson, D. B., & Clyman, S. G. (1999). A comparison of the generalizability of scores produced by expert raters and automated scoring systems. Applied Measurement in Education, 12, 281-299.
8. Clyman, S. G., Melnick, D. E., & Clauser, B. E. (1999). Computer-based case simulations from medicine: Assessing skills in patient management. In A. Tekian, C. H. McGuire, & W. C. McGahie (Eds.), Innovative simulations for assessing professional competence (pp. 29-41). Chicago: University of Illinois, Department of Medical Education.
9. Coffman, W. E. (1971). Essay examinations. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 271-302). Washington, DC: American Council on Education.
10. Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.
11. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
12. Cronbach, L. J., Schonemann, P., & McKie, D. (1965). Alpha coefficients for stratified-parallel tests. Educational and Psychological Measurement, 25, 291-312.
13. Federation of State Medical Boards of the U. S., Inc. (FSMB), & National Board of Medical Examiners (NBME). (1999). Step 3 general instructions, content description, and sample items. Philadelphia: FSMB and NBME.
14. Fitzpatrick, A. R. (2008). NCME 2008 presidential address: The impact of anchor test configuration on student proficiency rates. Educational Measurement: Issues and Practice, Win, 2008, 34-40.
15. Hartog, P., & Rhodes, E. C. (1936). The marks of examiners. London: MacMillan and Co.
16. Jarjoura, D., & Brennan, R. L. (1982). A variance components model for measurement procedures associated with a table of specification. Applied Psychological measurement, 6, 161-171.
17. Jarjoura, D., & Brennan, R. L. (1983). Multivariate generalizability analysis for tests developed from tables of specification. In L. J. Fyans (Ed.), Generalizability theory: Inferences and practical applications. San Francisco: Jossey-Bass Inc.
18. Kane, M. (1997). Model-based practice analysis and test specifications. Applied Measurement in Education, 10(1), 5-18.
19. Linn, R. L., & Burton, E. (1994). Performance-based assessment: Implications of task specificity. Educational Measurement: Issues and Practice, 13, 5-8, 15.
20. Margolis, M. J., & Clauser, B. E. (2006). A regression-based procedure for automated scoring of a complex medical performance assessment. In D. M. Williamson, R. J. Mislevy, & I. I. Bejar (Eds.), Automated scoring for complex tasks in computer-based testing (pp. 123-167). Hillsdale, NJ: Lawrence Erlbaum Associates.
21. Margolis, M. J., Clauser, B. E., Cuddy, M. M., Ciccone, A., Mee, J., Harik, P., et al. (2006). Use of the Mini-CEX to rate examinee performance on a multiple-station clinical skills examination: A validity study. Academic Medicine (RIME Supplement), 81, S56-S60.
22. Raymond, M. R. (2001). Job analysis and the specification of content for licensure and certification examinations. Applied Measurement in Education, 14(4), 369-415.
23. Shavelson, R. J., Ruiz-Primo, M. A., & Wiley, E. W. (1999). Note on sources of sampling variability in science performance assessments. Journal of Educational Measurement, 36, 61-71.
24. Stalnaker, J. M. (1951). The essay type of examination. In E. F. Lindquist (Ed.), Educational measurement (pp. 495-530). Washington, DC: American Council on Education.
25. Swanson, D. B., Norman, G., & Linn, R. (1995). Performance-based assessment: Lessons from the health professions. Educational Researcher, 24, 5-11, 35.