The impact of examinee performance information on judges' cut scores in modified angoff standard-setting exercises

Educational Measurement: Issues and Practice

Volumn 33, Issue 1, 2014, Pages 15-22

  Margolis, Melissa J ^a   Clauser, Brian E ^a  

^a NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

Author keywords

Modified angoff; Standard setting

Indexed keywords

EID: 84897840832     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/emip.12025     Document Type: Article

References (20)

1. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
2. Brandon, P. R. (2004). Conclusions about frequently studied modified Angoff standard-setting topics. Applied Measurement in Education, 17, 59-88.
3. Busch, J. C., & Jaeger, R. M. (1990). Influence of type of judge, normative information, and discussion on standards recommended for the National Teacher Examinations. Journal of Educational Measurement, 27, 145-163.
4. Cizek, G. J. (2012). Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum.
5. Clauser, B. E., Harik, P., Margolis, M. J., McManus, I. C., Mollon, J., Chis, L., & Williams, S. (2009). Empirical evidence for the evaluation of performance standards estimated using the Angoff procedure. Applied Measurement in Education, 22(1), 1-21.
6. Clauser, B. E., Mee, J., Baldwin, S. G., Margolis, M. J., and Dillon, G. F. (2009). Judges' use of examinee performance data in an Angoff standard-setting exercise for a medical licensing examination: An experimental study. Journal of Educational Measurement, 46, 390-407.
7. Clauser, B. E., Mee, J., & Margolis, M. J. (2013). The effect of data format on integration of performance data into Angoff judgments. International Journal of Testing, 13, 65-85.
8. Clauser, B. E., Swanson, D. B., & Harik, P. (2002). A multivariate generalizability analysis of the impact of training and examinee performance information on judgments made in an Angoff-style standard-setting procedure. Journal of Educational Measurement, 39, 269-290.
9. Clauser, J., & Clauser, B. (2012, April). An examination of the replicability of Angoff standard setting results within a generalizability theory framework. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, BC, Canada.
10. Cross, L. H., Impara, J. C., Frary, R. B. & Jaeger, R. M. (1984). A comparison of three methods for establishing minimum standards on the National Teachers Examination. Journal of Educational Measurement, 21, 113-129.
11. Hambleton, R. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. Cizek (Ed.) Standard setting: Concepts, methods and perspectives (pp. 159-173). Mahwah, NJ: Lawrence Erlbaum.
12. Hurtz, G. M., & Auerbach, M. A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63, 584-601.
13. Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527-535.
14. Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
15. Kane, M. (2001). So much remains the same: Conception and status of validation in setting standards. In G. Cizek (Ed.) Standard setting: Concepts, methods and perspectives (pp. 53-88). Mahwah, NJ: Lawrence Erlbaum.
16. Kane, M.T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4 th ed., pp. 17-64). Westport, CT: American Council on Education/Praeger.
17. Mee, J., Clauser, B. E., & Margolis, M. J. (2013). The impact of process instructions on judges' use of examinee performance data in Angoff standard setting exercises. Educational Measurement: Issues and Practice, 32(3), 27-35.
18. Plake, B. S. (2008). Standard setters: Stand up and take a stand! Educational Measurement: Issues and Practice, 27(1), 3-9.
19. Plake, B. S., Impara, J., & Potenza, M. (1994). Content specificity of expert judges in a standard setting study. Journal of Educational Measurement, 31, 339-347.
20. Truxillo, D. M., Donahue, L. M., & Sulzer, J. L. (1996). Setting cutoff scores for personnel selection tests: Issues, illustrations, and recommendations. Human Performance, 9, 275-295.