How many raters should be used for establishing cutoff scores with the angoff method? A generalizability theory study

Educational and Psychological Measurement

Volumn 59, Issue 6, 1999, Pages 885-897

  Hurtz, Gregory M ^a   Hertz, Norman R ^a,b  

^a Department of Consumer Affairs ^*  (United States)

^b Office of Examination Resources   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0033240364     PISSN: 00131644     EISSN: None     Source Type: Journal    
DOI: 10.1177/00131649921970233     Document Type: Article

References (24)

1. Berk, R. A. (1986). A consumer's guide to setting performance standards on criterion-referenced tests. Review of Educational Research, 56, 137-172.
2. Brennan, R. L. (1992). Elements of generalizability theory. Iowa City, IA: ACT.
3. Brennan, R. L., Gao, X., & Colton, D. A. (1995). Generalizability analyses of work keys listening and writing tests. Educational and Psychological Measurement, 55, 157-176.
4. Brennan, R. L., & Lockwood, R. E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.
5. Busch, J. C., & Jaeger, R. M. (1990). Influence of type of judge, normative information, and discussion on standards recommended for the national teacher examinations. Journal of Educational Measurement, 27, 145-163.
6. Cascio, W. F., Alexander, R. A., & Barrett, G. V. (1988). Setting cutoff scores: Legal, psychometric, and professional issues and guidelines. Personnel Psychology, 41, 1-24.
7. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley.
8. Cross, L. H., Impara, J. C., Frary, R. B., & Jaeger, R. M. (1984). A comparison of three methods for establishing minimum standards on the National Teacher Examinations. Journal of Educational Measurement, 21, 113-129.
9. Fehrmann, M. L., & Woehr, D. J. (1990, April). Standard setting: The impact of rater training. Paper presented at the annual convention of the Society for Industrial and Organizational Psychology, Miami Beach, FL.
10. Fehrmann, M. L., Woehr, D. J., & Arthur, W., Jr. (1991). The Angoff cutoff score method: The impact of frame-of-reference rater training. Educational and Psychological Measurement, 51, 857-872.
11. Jarjoura, D., & Brennan, R. L. (1983). Multivariate generalizability models for tests developed from tables of specifications. In L. J. Fyans (Ed.), Generalizability theory: Inferences and practical applications. San Francisco: Jossey-Bass.
12. Kane, M. T. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
13. Livingston, S. A., & Zieky, M. J. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.
14. Maurer, T. J., & Alexander, R. A. (1992). Methods of improving employment test critical scores derived by judging test content: A review and critique. Personnel Psychology, 45, 727-762.
15. Maurer, T. J., Alexander, R. A., Callahan, C. M., Bailey, J. J., & Dambrot, F. H. (1991). Methodological and psychometric issues in setting cutoff scores using the Angoff method. Personnel Psychology, 44, 235-262.
16. Mehrens, W. A., & Popham, W. J. (1992). How to evaluate the legal defensibility of high-stakes tests. Applied Measurement in Education, 5, 265-283.
17. Norcini, J. J., Lipner, R. S., Langdon, L. O., & Strecker, C. A. (1987). A comparison of three variations on a standard-setting method. Journal of Educational Measurement, 24, 56-64.
18. Norcini, J., Shea, J., & Grosso, L. (1991). The effect of numbers of experts and common items on cutting score equivalents based on expert judgment. Applied Psychological Measurement, 15, 241-246.
19. Norcini, J. J., Shea, J. A., & Kanya, D. T. (1988). The effect of various factors on standard setting. Journal of Educational Measurement, 25, 57-65.
20. Plake, B. S., Impara, J. C., & Potenza, M. T. (1994) Content specificity of expert judgments in a standard-setting study. Journal of Educational Measurement, 31, 339-347.
21. Reilly, R. R., Zink, D. L., & Israelski, E. W. (1984). Comparison of direct and indirect methods for setting minimum passing scores. Applied Psychological Measurement, 8, 421-429.
22. Shavelson, R. J., & Webb, N. M. (1991). Generalizibility theory: A primer. Newbury Park, CA: Sage.
23. Shepard, L. (1980). Standard setting issues and methods. Applied Psychological Measurement, 4, 447-467.
24. Van der Linden, W. J. (1982). A latent trait method for determining intrajudge inconsistency in the Angoff and Nedelsky techniques of standard setting. Journal of Educational Measurement, 19, 295-308.