The Effect of Data Format on Integration of Performance Data Into Angoff Judgments

International Journal of Testing

Volumn 13, Issue 1, 2013, Pages 65-85

  Clauser, Brian E ^a   Mee, Janet ^a   Margolis, Melissa J ^a  

^a NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

Author keywords

Angoff procedure; credentialing; high stakes testing; performance data; standard setting; validity

Indexed keywords

EID: 84871884171     PISSN: 15305058     EISSN: 15327574     Source Type: Journal    
DOI: 10.1080/15305058.2012.720316     Document Type: Article

References (27)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. 1999. Standards for educational and psychological tests, Washington, DC: American Educational Research Association.
2. Angoff, W. H. 1971. "Scales, norms, and equivalent scores". In Educational measurement (2nd ed., Edited by: Thorndike, R. L. 508-600. Washington, DC: American Council on Education.
3. Boursicot, K. A. M., Roberts, T. E. and Pell, G. 2006. Standard setting for clinical competence at graduation from medical school: A comparison of passing scores across five medical schools. Advances in Health Sciences Education, 11: 173-183.
4. Busch, J. C. and Jaeger, R. M. 1990. Influence of type of judge, normative information, and discussion on standards recommended for the National Teacher Examinations. Journal of Educational Measurement, 27: 145-163.
5. Clauser, B. E., Harik, P., Margolis, M. J., McManus, I. C., Mollon, J., Chis, L. and Williams, S. 2009. Empirical evidence for the evaluation of performance standards estimated using the Angoff procedure. Applied Measurement in Education, 22,: 1-21.
6. Clauser, B. E., Mee, J., Baldwin, S. G., Margolis, M. J. and Dillon, G. F. 2009. Judges' use of examinee performance data in an Angoff standard-setting exercise for a medical licensing examination: An experimental study. Journal of Educational Measurement, 46(4): 390-407.
7. Clauser, B. E., Swanson, D. B. and Harik, P. 2002. A multivariate generalizability analysis of the impact of training and examinee performance information on judgments made in an Angoff-style standard-setting procedure. Journal of Educational Measurement, 39: 269-290.
8. Cross, L. H., Impara, J. C., Frary, R. B. and Jaeger, R. M. 1984. A comparison of three methods for establishing minimum standards on the National Teacher Examinations. Journal of Educational Measurement, 21: 113-129.
9. Davis-Becker, S. L., Buckendahl, C. W. and Gerrow, J. 2011. Evaluating the bookmark standard setting method: The impact of random item ordering. International Journal of Testing, 11: 24-37.
10. Dillon, G. F. 1996. The expectations of standard setting judges. CLEAR Exam Review, 7(2): 22-25.
11. Goodwin, L. D. 1999. Relations between observed item difficulty levels and Angoff minimum passing levels for a group of borderline candidates. Applied Measurement in Education, 12(1): 13-28.
12. Hambleton, R. K. 2001. "Setting performance standards on educational assessments and criteria for evaluating the process". In Standard setting: Concepts, methods and perspectives, Edited by: Cizek, G. 159-173. Mahwah, NJ: Erlbaum Associates.
13. Hambleton, R. K., Brennan, R. L., Brown, W., Dodd, B., Forsyth, R. A., Mehrens, W. A. and... Zwick, R. 2000. A response to "Setting reasonable and useful performance standards" in the National Academy of Sciences' grading the nation's report card. Educational Measurement: Issues and Practice, 19(2): 5-14.
14. Hambleton, R. K. and Pitoniak, M. J. 2006. "Setting performance standards". In Educational measurement (4th ed., Edited by: Brennan, R. L. 433-470. Westport, CT: American Council on Education/Praeger.
15. Hambleton, R. K., Swaminathan, H. and Jane Rogers, H. 1991. MMSS Fundamentals of item response theory, Newbury Park, CA: Sage.
16. Hurtz, G. M. and Auerbach, M. A. 2003. A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63: 584-601.
17. Kane, M. 1994. Validating the performance standards associated with passing scores. Review of Educational Research, 64: 425-461.
18. Maurer, T. J. and Alexander, R. A. 1992. Methods of improving employment test critical scores derived by judging test content: A review and critique. Personnel Psychology, 45: 727-762.
19. Maurer, T. J., Alexander, R. A., Callahan, C. M., Bailey, J. J. and Dambrot, F. H. 1991. Methodological and psychometric issues in setting cutoff scores using the Angoff method. Personnel Psychology, 44: 235-262.
20. Norcini, J. J., Shea, J. A. and Kanya, D. T. 1988. The effects of various factors on standard setting. Journal of Educational Measurement, 25: 57-65.
21. Novakovic, N. 2008. Thinking on the edge: The influence of discussion and statistical data on awarders' perceptions of borderline candidates in an Angoff awarding meeting. International Journal of Training Research, 6: 74-100.
22. Papageorgiou, S. 2010. Investigating the decision-making process of standard setting participants. Language Testing, 27: 261-282.
23. Plake, B. S., Impara, J. and Potenza, M. 1994. Content specificity of expert judges in a standard setting study. Journal of Educational Measurement, 31: 339-347.
24. Raymond, M. R. and Reid, J. B. 2001. "Who made thee a judge: Selecting and training participants for standard setting". In Standard setting: Concepts, methods, and perspectives, Edited by: Cizek, G. 119-157. Mahwah, NJ: Lawrence Erlbaum Associates.
25. Reckase, M. D. 2001. "Innovative methods for helping standard setting participants to perform their task: The role of feedback regarding consistency, accuracy, and impact". In Standard setting: Concepts, methods and perspectives, Edited by: Cizek, G. 159-173. Mahwah, NJ: Erlbaum Associates.
26. Shepard, L. A. 1995. "Implications for standard setting of the National Academy of Educational Evaluation of the National Assessment of Educational Progress achievement levels". In Proceedings of the joint conference on standard setting for large-scale assessments of the National Assessment Governing Board and the National Center for Educational Statistics, 143-159. Washington, DC: U.S. Government Printing Office.
27. Truxillo, D. M., Donahue, L. M. and Sulzer, J. L. 1996. Setting cutoff scores for personnel selection tests: Issues, illustrations, and recommendations. Human Performance, 9: 275-295.