The Bookmark Standard Setting Procedure

Setting Performance Standards Foundations, Methods, and Innovations

Volumn , Issue , 2012, Pages 225-254

  Lewis, Daniel M ^a   Mitzel, Howard C ^b   Mercado, Ricardo L ^a   Schulz, E Matthew ^b  

^a CTB MCGRAW HILL   (United States)

^b Pacific Metrics ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84876980993     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203848203     Document Type: Chapter

References (57)

1. Abel, R. L. (1962). Content standard test scores. Educational and Psychological Measurement, 22(1), 15–25.
2. ACT, Inc. (April, 2005a). Process report: Developing achievement levels on the 2005 National Assessment of Educational Progress in grade twelve mathematics. Iowa City, IA: Author.
3. ACT, Inc. (April, 2005b). Special studies report: Developing achievement levels on the 2005 National Assessment of Educational Progress in grade twelve mathematics. Iowa City, IA: Author.
4. ACT, Inc. (July, 2010). Developing Achievement Levels on the 2009 National Assessment of Educational Progress in Science for Grades Four, Eight, and Twelve. Iowa City, IA: Author.
5. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508–600). Washington, DC: American Council on Education.
6. Barton, P. E. (2009). Contents for a new NAEP report: The five largest states. Paper prepared for the National Assessment Governing Board. Retrieved January 1, 2011, from http://www.nagb.org/publications/reports-papers.htm
7. Bejar, I. I., Braun, H., & Tannenbaum, R. (2007). A prospective, predictive and progressive approach to standard setting. In R. W. Lissitz (Ed.), Assessing and modeling cognitive development in school: Intellectual growth and standard setting (pp. 1–30). Maple Grove: MN: Jam Press.
8. Bock, R. D., Mislevy, R., & Woodson, C. (1982). The next stage in educational assessment. Educational Researcher, 11(3), 4–11.
9. Bourque, M. L. (2009). A history of NAEP achievement levels: Issues, implementation, and impact, 1989–2009. Paper commissioned for the 20th anniversary of the National Assessment Governing Board. Retrieved December 16, 2010 from http://www.nagb.org/who-we-are/20-anniversary/bourque-achievement-levels-formatted.pdf
10. Cochran, W. G. (1963). Sampling techniques. New York: Wiley.
11. CTB Macmillan/McGraw-Hill. (1992, June). Final technical report: Maryland School Performance Assessment Program 1991. Baltimore, MD: Maryland State Department of Education.
12. Engelhard, Jr., G., & Gordon, B. (2000). Setting and evaluating performance standards for high stakes writing assessments. In M. Wilson & G. Engelhard, Jr. (Eds.), Objective measurement: Theory into practice (Vol. 5. pp, 3–14). Stamford, CT: Ablex.
13. Educational Testing Service. (1987). The NAEP 1983–84 technical report. Princeton, NJ: Author.
14. Ferrara, S., Lewis, D. M., Mercado, R., & Egan, K. (2011, February). A method for setting nationally and internationally benchmarked performance standards: Proficiency level desciptors, procedures, and results. Presentation at a meeting of the Comprehensive Assessment System state collaborative of the Council of Chief State School Officers, Atlanta, GA.
15. Green, D. R., Trimble, C. S., & Lewis, D. L. (2003). Interpreting the results of three different standard setting procedures. Educational Measurement: Issues and Practice, 22(1), 22–32.
16. Grosse, M. E., & Wright, B. D. (1986). Setting, evaluating, and maintaining certification standards with the Rasch model. Evaluation and the Health Professions, 9, 267–285.
17. Hambleton, R. K. (2001) Setting performance standards on educational assessments and criteria for evaluating the process. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 89–116). Mahwah, NJ: Erlbaum.
18. Hambleton, R., & Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational Measurement, fourth edition (pp. 433–470). New York: Praeger.
19. Hambleton, R., & Zenisky, A. (2010, June). Improving the ways we report test scores. Presentation at the annual meeting of the Canadian Educational Research Association. Retrieved December 22, 2010, from http://www.mea.uottawa.ca/documents/ReportingRH.ppt
20. Huynh, H. (1994, October). Some technical aspects in standard setting. In Proceedings of the joint conference on standard setting for large scale assessment programs (pp. 75–91). Washington, DC: National Assessment Governing Board and National Center for Education Statistics.
21. Huynh, H. (2006). A clarification on the response probability criterion RP67 for standard settings based on Bookmark and item mapping. Educational Measurement: Issues and Practice, 25(2), 19–20.
22. Jaeger, R., & Mills, C. (1998, April). An integrated judgment procedure for setting standards on complex large-scale assessments. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
23. Karantonis, A., & Sireci, S. G. (2006). The Bookmark standard setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4–12.
24. Lewis, D. M., Mitzel, H. C., Green, D. R., & Patz, R. J. (1999, April). The (unmodified) Bookmark standard setting procedure. Mini course presented at the annual meetings of the American Educational Research Association and National Council on Measurement in Education, Montreal, Canada.
25. Lewis, D. M., & Green, D. R. (1997, June). The validity of Performance Level Descriptors. Paper presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
26. Lewis, D. M., Green, D. R., Mitzel, H. C., Baum, K., & Patz, R. J. (1998, April). The Bookmark standard setting procedure: Methodology and recent implementations. Paper presented at the annual meeting of the National Council for Measurement in Education, San Diego, CA.
27. Lewis, D. M., & Haug, C. (2005). Aligning policy and methodology to achieve consistent across-grade performance standards. Applied Measurement in Education, 18(1), 11–34.
28. Lewis, D. M., & Mitzel, H. C. (1995, September). An item response theory based standard setting procedure. Symposium presented at the annual meeting of the California Educational Research Association, Lake Tahoe, NV.
29. Lewis, D. M., Mitzel, H. C., & Green, D. R. (1996, June). Standard setting: A Bookmark approach. Symposium presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
30. Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980. In D. Kahneman & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 306–334). London: Cambridge University Press.
31. Masters, G. N., Adams, R., & Loken, J. (1994). Mapping student achievement. International Journal of Educational Research, 21, 595–609.
32. Mills, C. N., & Jaeger, R. J. (1998). Creating descriptions of desired student achievement when setting performance standards. In L. Hansche (Ed.), Handbook for the development of performance standards (pp. 73–85). Washington, DC: US Department of Education and the Council of Chief State School Officers.
33. Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2000). The Bookmark procedure: Cognitive perspectives on standard setting. In G. J. Cizek, (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 249–281). Mahwah, NJ: Erlbaum.
34. National Academies of Sciences. (2005). Measuring literacy: Performance levels for adults, interim report. Washington, DC: Author.
35. No Child Left Behind (NCLB). (2002). Public Law 107-110. 10th Congress.
36. Pacifi c Metrics Corporation. (2009). Operational Algebra I end-of-course test standard setting report; A report submitted to the Louisiana Department of Education. Monterey, CA: Author.
37. Pacifi c Metrics Corporation. (2010). Operational English II end-of-course test standard setting report: Report submitted to the Louisiana Department of Education. Monterey, CA: Author.
38. Pacifi c Metrics Corporation. (2011). Operational Geometry I end-of-course test standard setting report: Report submitted to the Louisiana Department of Education. Monterey, CA: Author.
39. Pashley, P. J., & G. W. Phillips (1993). Toward world class standards: A research study linking national and international assessments. Princeton, NJ: Educational Testing Service.
40. Perie, M. (2008). A guide to understanding and developing performance-level descriptors. Educational Measurement: Issues and Practice, 27(4), 15–29.
41. Philips, G. W., Mullis, I. V. S., Bourque, M. L., Williams, P. L., Hambleton, R. K., Owen, E. H., & Barton, P. E. (1993). Interpreting NAEP scales. Washington, DC: U.S. Department of Education, National Center for Education Statistics.
42. Philips, G. W. (2010). International benchmarking: State education performance standards. Washington DC: American Institutes for Research.
43. Schulz, E. M. (2006). Commentary: A response to Reckase’s conceptual framework and examples for evaluating standard setting methods. Educational Measurement: Issues and Practice, 25(2), 4–13.
44. Schulz, E. M. & Mitzel, H. C. (2005, April). Th e Mapmark standard setting method. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Quebec, Canada.
45. Schulz, E. M., & Mitzel, H. (2009). A Mapmark method of standard setting as implemented for the National Assessment Governing Board. In E. V. Smith, Jr., & G. E. Stone (Eds.), Applications of Rasch measurement in criterion referenced testing: Practice analysis to score reporting (pp. 194–235). Maple Grove, MN: JAM Press.
46. Shen, L. (2001, April). A comparison of Angoff and Rasch model based item map methods in standard setting. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.
47. Shepard, L., Glaser, R., Linn, R., & Bohrnstedt, G. (Eds.). (1993). Setting performance standards for student achievement Washington, DC: National Academy of Education.
48. Skaggs, G., Hein, S. F., & Awuor, R. (2007). Setting passing scores on passage-based tests: A comparison of traditional and single-passage Bookmark methods. Applied Measurement in Education, 20(4), 405-426.
49. Stone, G. E. (2001). Objective standard setting (or Truth in Advertising). Journal of Applied Measurement, 2(2), 187–201.
50. Thurstone, L. L. (1945). The prediction of choice. Psychometrika, X, 237–253.
51. Thurstone, L. L., (1927). The method of paired comparisons of social values. Journal of Abnormal and Social Psychology, 21, 384–400.
52. Thurstone, L. L. (1931). Infl uence of motion pictures on children’s attitudes. Journal of Social Psychology, 2, 291–305.
53. Wang, N. (2003). Use of the Rasch IRT model in standard setting: An item-mapping method. Journal of Educational Measurement, 40(3), 231–253.
54. Westat, Inc. (1994). Establishing proficiency levels and descriptions for the 1993 Maryland School Performance Assessment Program (MSPAP): Technical Report. Rockville, MD: Author.
55. Williams, N. J., & Schulz, E. M. (2005, April). An investigation of response probability (RP) values used in standard setting. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Montreal, Canada.
56. Yin, P., & Sconing, J. (2008) Estimating standard errors of cut scores for item rating and mapmark procedures: A generalizability theory approach. Educational and Psychological Measurement, 68(1), 25–41.
57. Zwick, R., Senturk, D., Wang, J., & Loomis, S. C. (2001). An investigation of alternative methods for item mapping in the National Assessment of Educational Progress. Educational Measurement: Issues and Practice, 20(2), 15–25.