The Impact of Process Instructions on Judges' Use of Examinee Performance Data in Angoff Standard Setting Exercises

Educational Measurement: Issues and Practice

Volumn 32, Issue 3, 2013, Pages 27-35

  Mee, Janet ^a   Clauser, Brian E ^a   Margolis, Melissa J ^a  

^a NATIONAL BOARD OF MEDICAL EXAMINERS   (United States)

Author keywords

Angoff method; Performance data; Raters; Standard setting

Indexed keywords

EID: 84884722418     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/emip.12013     Document Type: Article

References (26)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological tests. Washington, DC: American Educational Research Association.
2. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
3. Brennan, R. L. (1995). Standard setting from the perspective of generalizability theory. In M. L. Bourque (Ed.), Joint conference on standard setting for large-scale assessments (pp. 269-287). Washington, DC: NCSE-NAGB.
4. Busch, J. C., & Jaeger, R. M. (1990). Influence of type of judge, normative information, and discussion on standards recommended for the National Teacher Examinations. Journal of Educational Measurement, 27, 145-163.
5. Clauser, B. E., Harik, P., Margolis, M. J., McManus, I. C., Mollon, J., Chis, L., & Williams, S. (2009). An empirical examination of the impact of group discussion and examinee performance information on judgments made in the Angoff standard-setting procedure. Applied Measurement in Education, 22, 1-21.
6. Clauser, J. C., Margolis, M. J., & Clauser, B. E. (2012, April). An examination of the replicability of Angoff standard setting results within a Generalizability Theory framework. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Vancouver, BC.
7. Clauser, B. E., Mee, J., Baldwin, S. G., Margolis, M. J., & Dillon, G. F. (2009). Judges' use of examinee performance data in an Angoff standard-setting exercise for a medical licensing examination: An experimental study. Journal of Educational Measurement, 46, 390-407.
8. Clauser, B. E., Mee, J., & Margolis, M. J. (2013). The effect of data format on integration of performance data into Angoff judgments. International Journal of Testing, 13(1), 65-85.
9. Clauser, B. E., Swanson, D. B., & Harik, P. (2002). A multivariate generalizability analysis of the impact of training and examinee performance information on judgments made in an Angoff-style standard-setting procedure. Journal of Educational Measurement, 39, 269-290.
10. Davis-Becker, S. L., Buckendahl, C. W., & Gerrow, J. (2011). Evaluating the bookmark standard setting method: The impact of random item ording. International Journal of Testing, 11, 24-37.
11. Dillon, G. F. (1996). The expectations of standard setting judges. CLEAR Exam Review, 2, 22-26.
12. Dillon, G. F., & Walsh, W. P. (2000). Using performance data to set standards: Practical impact and the perception of judges. CLEAR Exam Review, 11(1), 15-18.
13. Goodwin, L. D. (1999). Relations between observed item difficulty levels and Angoff minimum passing levels for a group of borderline candidates. Applied Measurement in Education, 12(1), 13-28.
14. Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 433-470). Westport, CT: American Council on Education/Praeger.
15. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). MMSS fundamentals of item response theory. Newbury Park, CA: Sage.
16. Hurtz, G. M., & Auerbach, M. A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63, 584-601.
17. Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
18. Kane, M. T. (2001). So much remains the same: Conception and status of validation in setting standards. In G. Cizek (Ed.), Standard setting: Concepts, methods, and perspectives (pp. 53-88). Mahwah, NJ: Lawrence Erlbaum.
19. Margolis, M. J. (2011, April). The impact of examinee performance information on judges' cut scores in modified-Angoff standard setting exercises. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA.
20. Maurer, T. J., & Alexander, R. A. (1992). Methods of improving employment test critical scores derived by judging test content: A review and critique. Personnel Psychology, 45, 727-762.
21. Norcini, J. J., Shea, J. A., & Kanya, D. T., (1988). The effect of various factors on standard setting. Journal of Educational Measurement, 25, 57-65. doi:10.1111/j.1745-3984.1988.tb00291.x
22. Plake, B. S., Impara, J., & Potenza, M. (1994). Content specificity of expert judges in a standard setting study. Journal of Educational Measurement, 31, 339-347.
23. Raymond, M. R., & Reid, J. B. (2001). Who made thee a judge: Selecting and training participants for standard setting. In G. Cizek (Ed.), Standard setting: Concepts, methods, and perspectives (pp. 119-157). Mahwah, NJ: Lawrence Erlbaum.
24. Reckase, M. D. (2001). Innovative methods for helping standard setting participants to perform their task: The role of feedback regarding consistency, accuracy, and impact. In G. Cizek (Ed.) Standard setting: Concepts, methods and perspectives (pp. 159-173). Mahwah, NJ: Lawrence Erlbaum.
25. Shepard, L. A. (1995). Implications for standard setting of the National Academy of Education Evaluation of the National Assessment of Education Progress Achievement Levels. In Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 143-160). Washington, DC: National Assessment Governing Board and National Center for Educational Statistics.
26. Truxillo, D. M., Donahue, L. M., & Sulzer, J. L. (1996). Setting cutoff scores for personnel selection tests: Issues, illustrations, and recommendations. Human Performance, 9, 275-295.