Using a Difficulty-Anchored Rating Scale in Performing Angoff Ratings

International Journal of Selection and Assessment

Volumn 18, Issue 4, 2010, Pages 407-416

  Hoffman, Calvin C ^a   Tashima, C Chy ^b   Luck, Gypsi ^c  

^a ALLIANT INTERNATIONAL UNIVERSITY   (United States)

^b Los Angeles County Sheriff's Department   (United States)

^c CALIFORNIA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 78649582394     PISSN: 0965075X     EISSN: 14682389     Source Type: Journal    
DOI: 10.1111/j.1468-2389.2010.00522.x     Document Type: Article

References (31)

1. Angoff, W.H. (1971). Scales, norms, and equivalent scores. In Thorndike, R.L. (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
2. Biddle, R.E. (1993). How to set cutoff scores for knowledge tests used in promotion, training, certification, and licensing. Public Personnel Management, 22, 63-79.
3. Berk, R.A. (1986). A consumer's guide to setting performance standards on criterion referenced tests. Review of Educational Research, 56, 137-172.
4. Berk, R.A. (1996). Standard setting: The next generations (where few psychometricians have gone before!). Applied Measurement in Education, 9, 215-235.
5. Bowers, J.J. and Shindoll, R.R. (1989). A comparison of the Angoff, Beuk, and Hofstee methods for setting a passing score (ACT Research Report Series, 89-2). Iowa City, IA: American College Testing.
6. Busch, J.C. and Jaeger, R.M. (1990). Influence of type of judge, normative information, and discussion on standards recommended for the national teacher examinations. Journal of Educational Measurement, 27, 145-163.
7. Cascio, W.F., Alexander, R.A. and Barrett, G.V. (1988). Setting cutoff scores: Legal, psychometric, and professional issues and guidelines. Personnel Psychology, 41, 1-24.
8. Contreras v. City of Los Angeles656 F.2d 1267 (9th circular), 1981.
9. Coward, W.M. and Sackett, P.R. (1990). Linearity in ability-performance relationships: A reconfirmation. Journal of Applied Psychology, 75, 297-300.
10. Crocker, L. and Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth/Thompson Learning.
11. Ebel, R.L. (1972). Essentials of educational measurement. Englewood Cliffs, NJ: Prentice-Hall.
12. Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor & Justice. (1978). Uniform Guidelines on Employee Selection Procedures. Federal Register, 43, 38290-38315.
13. Fehrmann, M.L., Woehr, D.J. and Arthur, W. Jr (1991). The Angoff cutoff score method: The impact of frame-of-reference rater training. Educational and Psychological Measurement, 51, 857-872.
14. Garrido, M. and Payne, D. (1991). The effect of judges' knowledge of item data on two forms of the Angoff standard setting method. Georgia Educational Researcher, 8, 7-36.
15. Hambleton, R.K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In Cizek, G.J. (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 89-116). Mahwah, NJ: Lawrence Erlbaum Associates.
16. Hambleton, R.K. and Plake, B.S. (1995). Using an extended Angoff procedure to set standards on complex performance assessments. Applied Measurement in Education, 8, 41-55.
17. Hoffman, C.C. and Thornton, G.C. III (1997). Examining selection utility where competing predictors differ in adverse impact. Personnel Psychology, 50, 455-470.
18. Hurtz, G.M. and Auerbach, M.A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63, 584-601.
19. Impara, J.C. and Plake, B.S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353-366.
20. Jaeger, R. (1991). Selection of judges for standard setting. Educational Measurement Issues and Practice, 10, 3-14.
21. Jeanneret, P.R. and Stelly, D.J. (2003) Setting cutting scores using regression: Would you do it backwards? Master Tutorial presented at the Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL.
22. Kane, M.T. (1987). On the use of IRT methods with judgmental standard setting procedures. Journal of Educational Measurement, 24, 333-345.
23. Kehoe, J. and Olson, A. (2005). Cut scores and employment discrimination litigation. In Landy, F. (Ed.), Employment discrimination litigation (pp. 410-449). Society for Industrial and Organizational Psychology, San Francisco, CA: Jossey-Bass.
24. Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14, 3-19.
25. Plake, B.S., Impara, J.C. and Irwin, P.M. (2000). Consistency of Angoff-based predictions of item performance: Evidence of technical quality of results from the Angoff standard setting method. Journal of Educational Measurement, 37, 437-455.
26. Ployhart, R.E. and Holtz, B.C. (2008). The diversity-validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Personnel Psychology, 61, 153-172.
27. Ricker, K.L. (2003). Setting Cut Scores: Critical review of Angoff and modified-Angoff methods. Paper presented at the conference of the Canadian Society for the Study of Education, Halifax, Nova Scotia.
28. Taube, K.T. (1997). The incorporation of empirical item difficulty data in the Angoff standard-setting procedure. Evaluation and the Health Professions, 20, 479-498.
29. Society for Industrial and Organizational Psychology (SIOP). (2003). Principles for the validation and use of personnel selection procedures (4th ed.). College Park, MD: Author.
30. U.S. v. South Carolina, 434 US 1026, 1978.
31. Wheeler, P. (1991). The relationship between modified Angoff knowledge estimating judgments and item difficulty values for seven NTE specialty area tests. Paper presented at the annual meeting of the California Educational Research Association (ERIC Document Reproduction Service No. ED 340 745), San Diego, CA.