Essential Steps in Setting Performancestandards on Educational Tests and Strategies for Assessing the reliability of results

Setting Performance Standards Foundations, Methods, and Innovations

Volumn , Issue , 2012, Pages 47-76

  Hambleton, Ronald K ^a   Pitoniak, Mary J ^b   Copella, Jenna M ^b  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84875357981     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203848203     Document Type: Chapter

References (65)

1. ACT, Inc. (1997). Developing achievement levels on the 1998 NAEP in civics and writing: The design document. Iowa City, IA: Author.
2. ACT, Inc. (2005a, April). Developing achievement levels on the 2005 National Assessment of Educational Progress in grade twelve mathematics: Process report. Iowa City, IA: Author. Retrieved from http://www.nagb.org/publications/ achievement.htm
3. ACT, Inc. (2005b, May). Developing achievement levels on the 2005 National Assessment of Educational Progress in grade twelve mathematics: Technical report. Iowa City, IA: Author. Retrieved from http://www.nagb.org/publications/ achievement.htm
4. ACT, Inc. (2007a, June). Developing achievement levels on the 2006 National Assessment of Educational Progress in grade twelve economics: Process report. Iowa City, IA: Author. Retrieved from http://www.nagb.org/publications/ achievement.htm
5. ACT, Inc. (2007b, July). Developing achievement levels on the 2006 National Assessment of Educational Progress in grade twelve economics: Technical report. Iowa City, IA: Author. Retrieved from http://www.nagb.org/publications/ achievement.htm
6. ACT, Inc. (2009, January). Developing achievement levels on the 2009 National Assessment of Educational Progress for science at grades 4, 8, and 12: Design document. Iowa City, IA: Author. Retrieved from http://www.act.org/naepscience/ pdf/DesignDocument.pdf
7. American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, DC: AERA.
8. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508–600). Washington, DC: American Council on Education.
9. Bandeira de Mello, V., Blankenship, C., & McLaughlin, D. H. (2009). Mapping state profi ciency standards onto NAEP scales: 2005–2007 (NCES 2010-456). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.
10. Brennan, R. L. (2002). Estimated standard error of a mean when there are only two observations (CASMA Technical Note No. 1). Iowa City: University of Iowa, Center for Advanced Studies in Measurement and Assessments.
11. Camilli, G., Cizek, G. J., & Lugg, C. A. (2001). Psychometric theory and the validation of performance standards: History and future perspectives. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 445–475). Mahwah, NJ: Erlbaum.
12. Center on Education Policy. (2010, December). State high school tests: Exit exams and other assessments. Washington, DC: Author. Retrieved from http://www.cep-dc.org
13. Cizek, G. J. (1996a). Standard-setting guidelines. Educational Measurement: Issues and Practice, 15, 12–21.
14. Cizek, G. J. (1996b). Setting passing scores. Educational Measurement: Issues and Practice, 15, 20–31.
15. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.
16. Cizek, G. J., Bunch, M. B., & Koons, H. (2004). Setting performance standards: Contemporary methods. Educational Measurement: Issues and Practice, 23, 31–50.
17. Clauser, B. E., & Clyman, S. G. (1994). A contrasting-groups approach to standard setting for performance assessments of clinical skills. Academic Medicine, 69(10), S42–S44.
18. Clauser, B. E., Mee, J., Baldwin, S. G., Margolis, M. J., & Dillon, G. F. (2009). Judges’ use of examinee performance data in an Angoff standard-setting exercise for a medical licensing examination: An experimental study. Journal of Educational Measurement 46, 390–407.
19. Loomis, S. C., & Bourque, M. L. (2001). From tradition to innovation: Standard setting on the National Assessment of Educational Progress. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 175–217). Mahwah, NJ: Erlbaum.
20. Ebel, R. L. (1972). Essentials of educational measurement (2nd ed.). Englewood Cliff s, NJ: Prentice-Hall.
21. Geisinger, K. F. (1991). Using standard-setting data to establish cutoff scores. Educational Measurement: Issues and Practice, 10(2), 17–21.
22. Geisinger, K. F., & McCormick, C. M. (2010). Adopting cut scores: Post-standard-setting panel considerations for decision makers. Educational Measurement: Issues and Practice, 29(1), 38–44.
23. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65–110). Westport, CT: American Council on Education/Praeger.
24. Hambleton, R. K., & Bourque, M. L. (1991). The levels of mathematics achievement: Initial performance standards for the 1990 mathematics assessment (Technical Report, Vol. 3). Washington, DC: National Assessment Governing Board.
25. Hambleton, R. K., Brennan, R. L., Brown, W., Dodd, B., Forsyth, R. A., Mehrens, W. A., … Zwick, R. (2000). A response to “Setting Reasonable and Useful Performance Standards” in the National Academy of Sciences’ Grading the Nation’s Report Card.” Educational Measurement: Issues and Practice, 19, 5–13.
26. Hambleton, R. K., & Copella, J. (2011). Revalidation of the 2004 performance standard on the Uniform CPA Exam. Ewing, NJ: American Institute for Certified Public Accountants.
27. Hambleton, R. K., Jaeger, R. M., Plake, B. S., & Mills, C. N. (2000). Setting performance standards on complex educational assessments. Applied Psychological Measurement, 24(4), 355–366.
28. Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 433–470). Westport, CT: American Council on Education/Praeger.
29. Hambleton, R. K., & Plake, B. S. (1995). Using an extended Angoff procedure to set standards on complex performance assessments. Applied Measurement in Education, 8, 41–56.
30. Hambleton, R. K., & Powell, S. (1983). A framework for viewing the process of standard-setting. Evaluation and the Health Professions, 6, 3–24.
31. Hambleton, R. K., & Slater, S. C. (1995, October). Are NAEP executive summary reports understandable to policymakers and educators? Los Angeles, CA: CRESST, UCLA.
32. Huff, K., & Plake, B. S. (2010). Innovations in setting performance standards for K-12 test-based accountability. Measurement: Interdisciplinary Research and Perspectives, 8, 130–144.
33. Jaeger, R. M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 485–514). New York, NY: Macmillan.
34. Jaeger, R. M. (1991). Selection of judges for standard setting. Educational Measurement: Issues and Practice, 10, 3–6, 10.
35. Jaeger, R. M. (1995). Setting performance standards through two-stage judgmental policy capturing. Applied Measurement in Education, 8, 15–40.
36. Jaeger, R. M., & Mills, C. N. (2001). An integrated judgment procedure for setting standards on complex, large-scale assessments. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 313– 338). Mahwah, NJ: Erlbaum.
37. Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425–462.
38. Kane, M. (2001). So much remains the same: Conception and status of validation in setting standards. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 53–88). Mahwah, NJ: Erlbaum.
39. Karantonis, A., & Sireci, S. G. (2006). The Bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4–12.
40. Katz, I., Tannenbaum, R. J., & Kannan, P. (2009, Summer). Virtual standard setting. CLEAR Exam Review, 19–27.
41. Kingston, N. M., Kahl, S. R., Sweeney, K. R., & Bay, L. (2001). Setting performance standards using the body of work method. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 219–248). Mahwah, NJ: Erlbaum.
42. Linn, R. L., & Herman, J. L. (1997). A policymaker’s guide to standards-led assessment. Denver, CO: Education Commission of the States.
43. Measured Progress. (2011, February 8). Developing achievement levels on the National Assessment of Educational Progress for writing grades 8 and 12 in 2011 and grade 4 in 2013. Dover, NH: Author. Retrieved from http://www. wested.org
44. Mills, C. N., Hambleton, R. K., Biskin, B., Kobrin, J., Evans, J., & Pfeff er, M. (2000). A comparison of the standard-setting methods for the Uniform CPA Examination. Jersey City, NJ: American Institute for Certified Public Accountants.
45. Mills, C. N., & Jaeger, R. J. (1998). Creating descriptions of desired student achievement when setting performance standards. In L. Hansche (Ed.), Handbook for the development of performance standards (pp. 73–85). Washington, DC: US Department of Education and the Council of Chief State School Officers.
46. Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2001). The bookmark procedure: Cognitive perspectives on standard setting. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 249–281). Mahwah, NJ: Erlbaum.
47. Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14, 3–19.
48. Nichols, P., Twing, J., & Mueller, C. D. (2010). Standard-setting methods as measurement processes. Educational Measurement: Issues and Practice, 29(1), 14–24.
49. Pellegrino, J. W., Jones, L. R., & Mitchell, K. J. (Eds.). (1999). Grading the nation’s report card. Washington, DC: National Academy Press.
50. Perie, M. (2008). A guide to understanding and developing performance-level descriptors. Educational Measurement: Issues and Practice, 27(4), 15–29.
51. Pitoniak, M. J., & Morgan, D. L. (in press). Setting and validating cut scores for tests. In C. Secolsky (Ed.), Measurement, assessment, and evaluation in higher education. New York: Routledge.
52. Plake, B. S., & Hambleton, R. K. (2000). A standard setting method designed for complex performance assessments: Categorical assignments of student work. Educational Assessment, 6(3), 197–215.
53. Plake, B. S., & Hambleton, R. K. (2001). The analytic judgment method for setting standards on complex performance assessments. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 283–312). Mahwah, NJ: Erlbaum.
54. Plake, B. S., Hambleton, R. K., & Jaeger, R. M. (1997). A new standard-setting method for performance assessments: The dominant profile judgment method and some field-test results. Educational and Psychological Measurement, 57, 400–411.
55. Plake, B. S., Melican, G. J., & Mills, C. N. (1991). Factors infl uencing intrajudge consistency during standard-setting. Educational Measurement: Issues and Practice, 10, 15–16, 22–26.
56. Raymond, M. R., & Reid, J. B. (2001). Who made thee a judge? Selecting and training participants for standard setting. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 119–157). Mahwah, NJ: Erlbaum.
57. Reckase, M. D. (2001). Innovative methods for helping standard-setting participants to perform their task: The role of feedback regarding consistency, accuracy, and impact. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 159–173). Mahwah, NJ: Erlbaum.
58. Shepard, L., Glaser, R., Linn, R., & Bohrnstedt, G. (1993). Setting performance standards for achievement tests. Stanford, CA: National Academy of Education.
59. Sireci, S. G., Hambleton, R. K., & Pitoniak, M. J. (2004). Setting passing scores on licensure exams using direct consensus. CLEAR Exam Review, 15, 21–25.
60. Sireci, S. G., Hauger, J. B., Wells, C. S., Shea, C., & Zenisky, A. L. (2009). Evaluation of the standard setting on the 2005 grade 12 National Assessment of Educational Progress mathematics test. Applied Measurement in Education, 22, 339–358.
61. Skorupski, W., & Hambleton, R. K. (2005). What are panelists thinking when they participate in standard setting studies? Applied Measurement in Education, 18, 233–255.
62. van der Linden, W. J. (1982). A latent trait method for determining intrajudge inconsistency in the Angoff and Nedelsky techniques of standard-setting. Journal of Educational Measurement, 19(4), 295–308.
63. van der Linden, W. J. (1995). A conceptual analysis of standard-setting in large-scale assessments. In Proceedings of the joint NCES-NAGB conference on standard-setting for large-scale assessment (pp. 97–118). Washington, DC: U.S. Government Printing Office.
64. Zenisky, A., Hambleton, R. K., & Sireci, S. G. (2009). Getting the message out: An evaluation of NAEP score reporting practices with implications for disseminating test results. Applied Measurement in Education, 22(22), 359–375.
65. Zieky, M. J., Perie, M., & Livingston, S. A. (2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.