Estimating standard errors of cut scores for item rating and mapmark procedures: A generalizability theory approach

Educational and Psychological Measurement

Volumn 68, Issue 1, 2008, Pages 25-41

  Yin, Ping ^a,b   Sconing, James ^a  

^a ACT

^b NONE   (United States)

Author keywords

Generalizability theory; Item rating method; Mapmark method; Standard setting

Indexed keywords

EID: 37649015612     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164407301546     Document Type: Article

References (28)

1. Angoff, W.H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2 nd ed., pp. 508-600). Washington, DC: American Council on Education.
2. Berk, R.A. (1986). A consumer's guide to setting performance standards on criterion-referenced test. Review of Educational Research, 56, 137-172.
3. Bock, R.D. (1997). A brief history of item response theory. Educational Measurement: Issues and Practice, 16 (4). 21-33.
4. Brennan, R.L. (1995). Standard setting from the perspective of generalizability theory. In Proceedings of the joint conference on standard setting for large-scale assessments (Vol. 2). Washington, DC: National Center for Educational Statistics and National Assessment Governing Board.
5. Brennan, R.L. (2001 a). Generalizability theory. New York: Springer-Verlag.
6. Brennan, R.L. (2001 b). Manual for urGENOVA. Iowa City: University of Iowa, Iowa Testing Programs.
7. Brennan, R.L., & Lockwood, R.E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.
8. Cizek, G. J. (Eds.). (2001). Setting performance standards. Mahwah, NJ: Lawrence Erlbaum.
9. Council of Chief State School Officers (2001). State student assessment programs annual survey. Data Volume II. Washington, DC: Author.
10. Jaeger, R.M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (3 rd ed., pp. 485-514). Phoenix, AZ: Oryx Press.
11. Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
12. Kane, M., & Wilson, J. (1984). Errors of measurement and standard setting in mastery testing. Applied Psychological Measurement, 8, 425-461.
13. Hambleton, R.K., & Eignor, D.R. (1980). Competency test development, validation and standard setting. In R. M. Jaeger & C. K. Tittle (Eds.), Minimum competency achievement testing: Motives, models, measures and consequences (pp. 367-396). Berkeley, CA: McCutchan.
14. Hambleton, R., & Swaminathan, H. (1985). Item response theory: Principles and applications (pp. 61-69). Norwell, MA: Kluwer Academic Publishers.
15. Lee, G., & Lewis, D.M. (2001, April). A generalizability theory approach toward estimating standard errors of cutscores set using the bookmark standard setting procedure. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
16. Lewis, D.M., Mitzel, H.C., & Green, D.R. (1996, June). Standard setting: A bookmark approach. Paper presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
17. Loomis, S.C., & Bourque, M.L. (2001). From tradition to innovation: Standard setting on the National Assessment of Educational Progress. In G. J. Cizek (Ed.), Setting performance standards (pp. 249-281). Mahwah, NJ: Lawrence Erlbaum.
18. Loomis, S.C., & Hanick, P.L. (2000). Setting achievement levels on the 1998 National Assessment of Educational Progress in civics: Pilot study final report. Iowa City, IA: ACT, Inc.
19. Masters, G.N., Adams, R., & Lokan, J. (1994). Mapping student achievement. International Journal of Educational Research, 21, 595-609.
20. Mitzel, H.C., Lewis, D.M., Patz, R.J., & Green, D.R. (2001). The bookmark procedure: Psychological perspectives. In G. J. Cizek (Ed.), Setting performance standards (pp. 249-281). Mahwah, NJ: Lawrence Erlbaum.
21. Reckase, M.D. (2000). The evolution of the NAEP achievement levels setting process: A summary of the research and development efforts conducted by ACT. Iowa City, IA: ACT, Inc.
22. Schulz, E.M., Kolen, M., & Nicewander, W.A. (1999). A rationale for defining achievement levels using IRT-estimated domain scores. Applied Psychological Measurement, 23, 347-362.
23. Schulz, E.M., Lee, W., & Mullen, K. (2005). A domain-level approach to describing growth in achievement. Journal of Educational Measurement, 42, 1-26.
24. Schulz, E.M., & Mitzel, H. (2005, April). The mapmark standard setting method. Paper presented at the annual meeting of the National Council on Measurement and Education, Montreal, Canada.
25. Shepard, L. (1980). Standard setting, issues, and methods. Applied Psychological Measurement, 4, 447-467.
26. Surowiecki, J. (2004). The wisdom of crowds. New York: Doubleday.
27. Williams, N.J., & Schulz, E.M. (2005, April). An investigation of response probability (RP) values used in standard setting. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
28. Yin, P., & Schulz, E.M. (2005, April). A comparison of cut scores and cut score variability from Angoff-based and bookmark-based procedures in standard setting. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.