Commentary: A response to Reckase's conceptual framework and examples for evaluating standard setting methods

Educational Measurement: Issues and Practice

Volumn 25, Issue 3, 2006, Pages 4-13

  Schulz, E Matthew ^a,b  

^a ACT INC   (United States)

^b American College Testing Incorporated   (United States)

Author keywords

Bookmark method; Modified angoff method; Standard setting

Indexed keywords

EID: 33750989082     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2006.00061.x     Document Type: Conference Paper

References (33)

1. ACT, Inc. (April, 2005a). Developing achievement levels on the 2005 national assessment of educational progress in grade twelve mathematics; Process report. Iowa City, IA: Author.
2. ACT, Inc. (May, 2005b). Developing achievement levels on the 2005 national assessment of educational progress in grade twelve mathematics: Technical report. Iowa City, IA: Author.
3. ACT, Inc. (May, 2005c). Developing achievement levels on the 2005 national assessment of educational progress in grade twelve mathematics: Special Studies report. Iowa City, IA: Author
4. ACT, Inc. (May, 2005d). Developing achievement levels on the 2005 national assessment of educational progress in grade twelve mathematics: Domain development report. Iowa City, IA: Author.
5. American College Testing (1993). Setting achievement levels on the 1992 national assessment of educational progress inmathematics, reading, and writing. Iowa City, IA: Author.
6. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed.). Washington, DC: American Council on Education.
7. Egan, K. L., Lorié, W., & Brandstrom, A. J. (June, 2002). Setting standards by the bookmark: Some technical aspects of the bookmark standard setting procedure. Paper presented at the CCSSO National Conference on Large-Scale Assessment, Palm Springs, CA.
8. Glass, G. V. (1978). Standards and criteria. Journal of Educational Measurement, 15, 237-261.
9. Haertel, E. H. (2002). Standard setting as a participatory process. Educational Measurement: Issues and Practice, 21, 16-22.
10. th Annual National Roundtable Conference, Melbourne, Australia.
11. Humphry, S. M. (2005). Maintaining a common arbitrary unit in social measurement. Unpublished doctoral dissertation. Murdoch University, Perth, Western Australia. Available at http://wwwlib.murdoch.edu.au/adt/browse/view/ adt-MU20050830.95143.
12. Karantonis, A., & Sireci, S. G. (2006). The bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 35, 4-12.
13. Lewis, D. M., Mitzel, H. C., & Green, D. R. (June, 1996). Standard setting: A bookmark approach. In D.R. Green (Chair), IRT-based standard-setting procedures using behavioral anchoring. Symposium conducted at the Council of Chief State School Officers National Conference on Large Scale Assessment, Phoenix, AZ.
14. Loomis, S. C., & Bourque, M. L. (2001). From tradition to innovation: Standard setting on the national assessment of educational progress. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 175-217). Mahwah, NJ: Erlbaum.
15. Lorge, I., & Kruglov, L. (1953). The improvement of estimates of test difficulty. Educational and Psychological Measurement, 13, 34-46.
16. McGinty, D. (2005). Illuminating the black box of standard setting: An exploratory qualitative study. Applied Measurement in Education, 18, 269-287.
17. Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2001). The bookmark procedure: Psychological perspectives. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, & perspectives (pp. 249-281). Mahwah, NJ: Erlbaum.
18. Plake, B. S., Melican, G. J., & Mills, C. N. (1991). Factors influencing intrajudge consistency during standard setting. Educational Measurement: Issues and Practice, 10(2), 15-16, 22, 25.
19. Popham, W. J. (1978). As always, provocative. Journal of Educational Measurement, 15, 297-300.
20. Poulton, E. C. (1979). Models for biases in judging sensory magnitude. Psychological Bulletin, 86, 777-803.
21. Reckase, M. D., (2001). Innovative methods for helping standard setting panelists to perform their task: The role of feedback regarding consistency, accuracy, and impact. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 159-174). Mahwah, NJ: Erlbaum.
22. Reckase, M. D. (2006). A conceptual framework for a psychometric theory for standard setting with examples of its use for evaluating the functioning of two standard setting methods. Educational Measurement: Issues and Practice, 25(2), 4-18.
23. Schulz, E. M., Kolen, M., & Nicewander, W. A. (1999). A rationale for defining achievement levels using IRT-estimated domain scores. Applied Psychological Measurement, 23(4), 347-362.
24. Schulz, E. M., Lee, W., & Mullen, K. (2005). A domain-level approach to describing growth in achievement. Journal of Educational Measurement, 42, 1-26.
25. Schulz, E. M., & Mitzel, H. C. (April, 2005). The Mapmark standard setting method. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Quebec, Canada. ERIC Document ED490463.
26. Schulz, E. M., Perlman, C., Rice, W. K., & Wright, B. D. (1992). Vertically equating reading tests: An example from Chicago Public Schools. In M. Wilson (Ed.), Objective measurement: theory into practice, 1 (pp. 138-154). Norwood, NJ: Albex.
27. Shepard, L. (1980). Standard setting issues and methods. Applied Psychological Measurement, 4, 447-467.
28. Shepard, L. A. (October, 1994). Implications for standard setting of the National Academy evaluation of the National Assessment of Educational Progress achievement levels. Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments.
29. Shepard, L. A., Glaser, R., Linn, R. L., & Bohrnstedt, G. (1993). Setting performance standards for student achievement: A report of the National Academy of Education Panel of the evaluation of the NAEP trial state assessment: An evaluation of the 1992 achievement levels. Stanford, CA: Stanford University, National Academy of Education.
30. Surowiecki, J. (2004). The wisdom of crowds. New York, NY: Doubleday.
31. Williams, N., & Schulz, E. M. (March, 2005). An investigation of response probability values used in standard setting. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Montreal, Canada.
32. Yin, P., & Schulz, E. M. (March, 2005). A comparison of cut scores and cut score variability from Angoff-based and bookmark-based procedures in standardsetting. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Montreal, Canada.
33. Yin, P., & Sconing, J. (April, 2006). Estimating standard errors of cut scores forAngoff-based and bookmark-based procedures: A generalizability theory approach. Paper presented at the 2006 Annual Meeting of the National Council on Measurement in Education, San Francisco, CA.