Judges' Use of Examinee Performance Data in an Angoff Standard-Setting Exercise for a Medical Licensing Examination: An Experimental Study

Journal of Educational Measurement

Volumn 46, Issue 4, 2009, Pages 390-407

  Clauser, Brian E ^a   Mee, Janet ^a   Baldwin, Su G ^a   Margolis, Melissa J ^a   Dillon, Gerard F ^a  

^a NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 71549121966     PISSN: 00220655     EISSN: 17453984     Source Type: Journal    
DOI: 10.1111/j.1745-3984.2009.00089.x     Document Type: Article

References (20)

1. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed. Educational measurement (2nd ed., pp. 508 600). Washington, DC : American Council on Education.
2. Brandon, P. R. (2004). Conclusions about frequently studied modified Angoff standard setting topics. Applied Measurement in Education, 17, 59 88.
3. Busch, J. C., & Jaeger, R. M. (1990). Influence of type of judge, normative information, and discussion on standards recommended for the National Teacher Examinations. Journal of Educational Measurement, 27, 145 163.
4. Clauser, B. E., Harik, P., Margolis, M. J., McManus, I. C., Mollon, J., Chis, L., & Williams, S. (2009). Empirical evidence for the evaluation of performance standards estimated using the Angoff procedure. Applied Measurement in Education, 22, 1 21.
5. Clauser, B. E., Swanson, D. B., & Harik, P. (2002). A multivariate generalizability analysis of the impact of training and examinee performance information on judgments made in an Angoff-style standard-setting procedure. Journal of Educational Measurement, 39, 269 290.
6. Dillon, G. F. (1996). The expectations of standard setting judges. CLEAR Exam Review, 7 (2 22 25.
7. Dillon, G. F., & Walsh, W. P. (2000). Using performance data to set standards: Practical impact and the perception of judges. CLEAR Exam Review, 11 (1 15 18.
8. Goodwin, L. D. (1999). Relations between observed item difficulty levels and Angoff minimum passing levels for a group of borderline candidates. Applied Measurement in Education, 12, 13 28.
9. Hambleton, R. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. Cizek (Ed.) Standard setting: Concepts, methods and perspectives (pp. 89 116). Mahwah, NJ : Erlbaum Associates.
10. Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed. Educational measurement (4th ed., pp. 433 470). Westport, CT : American Council on Education/Praeger.
11. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA : Sage.
12. Hurtz, G. M., & Auerbach, M. A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63, 584 601.
13. Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425 461.
14. Maurer, T. J., & Alexander, R. A. (1992). Methods of improving employment test critical scores derived by judging test content: A review and critique. Personnel Psychology, 45, 727 762.
15. Plake, B. S., Impara, J. C., & Irwin, P. (1999, April). Validation of Angoff-based predictions of item performance. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
16. Raymond, M. R., & Reid, J. B. (2001). Who made thee a judge: Selecting and training participants for standard setting. In G. Cizek (Ed. Standard setting: Concepts, methods, and perspectives (pp. 119 157). Mahwah, NJ : Erlbaum Associates.
17. Reckase, M. D. (2001). Innovative methods for helping standard setting participants to perform their task: The role of feedback regarding consistency, accuracy, and impact. In G. Cizek (Ed. Standard setting: Concepts, methods and perspectives (pp. 159 173). Mahwah, NJ : Erlbaum Associates.
18. Shepard, L. A. (1995). Implications for standard setting of the National Academy of Educational Evaluation of the National Assessment of Educational Progress achievement levels. In Proceedings of the Joint Conference on standard setting for large-scale assessments of the National Assessment Governing Board and the National Center for Educational Statistics. Washington, DC : U.S. Government Printing Office, 143 159.
19. Smith, R. L., & Smith, J. K. (1988). Differential use of item information by judges using Angoff and Nedelsky procedures. Journal of Educational Measurement, 25, 259 274.
20. Truxillo, D. M., Donahue, L. M., & Sulzer, J. L. (1996). Setting cutoff scores for personnel selection tests: Issues, illustrations, and recommendations. Human Performance, 9, 275 295.