Performance Level Descriptors: History, Practice, and a Proposed Framework

Setting Performance Standards: Foundations, Methods, and Innovations

Volumn , Issue , 2012, Pages 79-106

  Egan, Karla L ^a   Schneider, M Christina ^a   Ferrara, Steve ^a  

^a CTB MCGRAW HILL   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84878121267     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203848203-12     Document Type: Chapter

References (67)

1. Aspen Systems Corporation. (1992). Survey of reactions to the use of achievement levels in reporting 1990 NAEP mathematics results, prepared under contract with the National Assessment Governing Board. Rockville, MD: Author.
2. Bejar, I. I., Braun, H. I., & Tannenbaum, R. J. (2007). A prospective, progressive, and predictive approach to standard setting. In R. Lissitz (Ed.), Assessing and modeling cognitive development in school (pp. 31-63). Maple Grove, MN: JAM Press.
3. Bourque, M. L. (2000, April). Setting student performance standards: Th e role of achievement level descriptions in the standard setting process. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
4. Bourque, M. L., & Byrd, S. (Eds.). (2000). Student performance standards on the National Assessment of Educational Progress: Affi rmation and improvement. Washington, DC: National Assessment Governing Board.
5. Brown, W. (2000). Reporting NAEP by achievement levels: An analysis of policy and external reviews. In M. L. Bourque & S. Byrd (Eds.), Student performance standards on the National Assessment of Educational Progress: Affi rmation and improvement (pp. 11-40). Washington, DC: National Assessment Governing Board.
6. Burstein, L., Koretz, D., Linn, R., Sugrue, B., Novak, J., Baker, E. L., & Harris, E. L. (1996). Describing performance standards: Validity of the 1992 National Assessment of Educational Progress PLDs as characterizations of mathematics performance. Educational Assessment, 3(1), 9-51.
7. Burt, W. M., & Stapleton, L. M. (2010). Connotative meanings of student performance labels used in standard setting. Educational Measurement: Issues and Practice, 29(4), 28-38.
8. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Th ousand Oaks, CA: Sage.
9. Council of Chief State School Offi cers (CCSSO) & National Governor’s Association (NGA). (2011). Common core state standards. Retrieved Feb 2, 2011, from http://www.corestandards.org/
10. Crane, E. W., & Winter, P. C. (2006). Setting coherent performance standards. Washington DC: Chief Council of State School Offi cers.
11. Cross, L. H., Impara, J. C., Frary, R. B., & Jaeger, R. M. (1984). A comparison of three methods for establishing minimum standards on the national teacher examinations. Journal of Educational Measurement, 21(2), 113-129.
12. Egan, K. L., Schneider, M. C., & Ferrara, S. (2011). Th e 6D framework: A validity framework for defi ning profi cient performance and setting cut scores for accessible tests. In S. Elliott, R. Kettler, P. Beddow, & A. Kurz (Eds.), Accessible tests of student achievement: Issues, innovations, and applications (pp. 275-292). New York, NY: Springer.
13. Ercikan, K., & Julian, M. (2002). Classifi cation accuracy of assigning student performance to performance levels: Guidelines for assessment development. Applied Measurement in Education, 15(3), 269-294.
14. Embretson, S. E., & Gorin, J. S. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38(4), 343-368.
15. Fehrmann, M. L., Woehr, D. J., & Arthur, Jr., W. (1991). Th e Angoff cutoff score method: Th e impact of frame-ofreference rater training. Educational and Psychological Measurement, 51, 857-887.
16. Ferrara, S., Swaffi eld, S., & Mueller, L. (2009). Conceptualizing and setting performance standards for alternate assessments. In W. D. Schafer & R. W. Lissitz (Eds.), Alternate assessments based on alternate achievement standards: Policy, practice, and potential (pp. 93-111). Baltimore, MD: Paul Brookes.
17. Ferrara, S., Svetina, D., Skucha, S., & Murphy, A. (2009). Test design with performance standards and achievement growth in mind. Manuscript submitted for publication.
18. Giraud, G., Impara, J. C., & Plake, B. S. (2005). A qualitative examination of teachers’ conception of the target examinee. Applied Measurement in Education, 18(3), 223-232.
19. Gorin, J. S. (2006). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25(4), 21-35.
20. Gorth, W. P., & Perkins, M. P. (1979). A study of minimum competency testing programs. Final program development resource document. Amherst, MA: National Evaluation Systems.
21. Haertel, E. H. (1999). Validity arguments for high-stakes testing:. In search of the evidence., Educational Measurement: Issues and Practice, 18(4), 5-9.
22. Hambleton, R. K., & Slater, S. (1997). Are NAEP executive summary reports understandable to policy makers and educators? (CSE Technical Report 430). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Teaching.
23. Hambleton, R. K., & Pitoniak, M. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 433-470). Westport, CT: Praeger.
24. Hansche, L. N. (Ed.). (1998). Meeting the requirements of Title I: Handbook for the development of performance standards. Washington, DC: U.S. Department of Education.
25. Hendrickson, A., Huff, K., & Luecht, R. (2010). Claims, evidence and achievement level descriptors as a foundation for item design and test specifi cations. Applied Measurement in Education, 23(4), 358-377.
26. Huff, K., & Plake, B. S. (2010). Evidence-centered assessment design in practice. Applied Measurement in Education, 23(4), 307-309.
27. Hurtz, G. M., & Auerbach, M. A. (2003). A meta-analysis of the eff ects of modifi cations to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63, 584-601.
28. Impara, J. C., & Plake, B. S. (1998). Teachers’ ability to estimate item diffi culty: A test of the assumptions in the Angoff standard setting method. Journal of Educational Measurement, 35, 69-81.
29. Impara, J. C., Giraud, G., Plake, B. S. (2000, Aprils). Th e infl uence of providing target group descriptors when setting a passing score. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
30. Jaeger, R. M. (1978, Spring). A proposal for setting a standard on the North Carolina High School Competency Test. Paper presented at the spring meeting of the North Carolina Association for Research in Education, Chapel Hill.
31. Jaeger, R. M. (1998). Reporting the results of the National Assessment of Educational Progress. Report commissioned by the NAEP Validity Studies (NVS) Panel. Retrieved January 2, 2011, from http://www.air.org/fi les/Jaeger_Reporting_ NAEP.pdf
32. Kane, M. T. (1994). Validating interpretive arguments for licensure and certifi cation examinations. Evaluation and the Health Professions, 17, 133-159.
33. Koretz, D., & Deibert, E. (1993). Interpretations of National Assessment of Educational Progress (NAEP) anchor points and achievement levels by the print media in 1991. Santa Monica, CA: RAND.
34. Leighton, J. P., & Gierl, M. J. (Eds.). (2007a). Cognitive diagnostic assessment for education: Th eory and practices. New York, NY: Cambridge University Press.
35. Leighton, J. P., & Gierl, M. J. (2007b). Defi ning and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 26, 3-16.
36. Lewis, D. M., & Green, R. (1997, Junes). Th e validity of PLDs. Paper presented at the National Conference on Large Scale Assessment, Colorado Springs, CO.
37. Linn, R. L. (1998). Partitioning responsibility for the evaluation of the consequences of assessment programs. Educational Measurement: Issues and Practice, 17, 28-30.
38. Mercado, R. L., & Egan, K. L. (2005, April). Performance level descriptors. Paper presented at the National Council of Measurement in Education, Montréal, Quebec.
39. Mehrens, W. A., & Lehmann, I. J. (1991). Measurement and evaluation in education and psychology. Fort Worth, TX: Harcourt Brace.
40. Messick, S. (1994). Th e interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23, 13-23.
41. Mills, C. N., & Jaeger, R. M. (1998). Creating descriptions of desired student achievement when setting performance standards. In L. N. Hansche (Ed.), Meeting the requirements of Title I: Handbook for the development of performance standards (pp. 73-85). Washington, DC: U.S. Department of Education.
42. Mills, C. N., Melican, G. J., & Ahluwalia, N. T. (1991). Defi ning minimal competence. Educational Measurement: Issues and Practice, 10(2), 7-10.
43. Mislevy, R. J., Steinberg, L. S., & Almond, R.G. (2002). On the role of task model variables in assessment design. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 97-128). Mahwah, NJ: Erlbaum.
44. Monroe, W. S., DeVoss, A. M., & Kelly, F. J. (1924). Educational tests and measurements. Cambridge, MA: Riverside.
45. Nassif, P. M. (1979). Setting standards. In W. P. Gorth & M. P. Perkins (Eds.), A study of minimum competency testing programs. Final program development resource document (pp. 93-123). Amherst, MA: National Evaluation Systems.
46. Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14, 3-19.
47. Nickerson, R. S. (2004). Cognition and chance: Th e psychology of probabilistic reasoning. Mahwah, NJ: Erlbaum.
48. No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002).
49. Norcini, J., Lipner, R. S., Langdon, L. O., & Strecker, C. A. (1987). A comparison of three variations on a standardsetting method. Journal of Educational Measurement, 24, 56-64.
50. Norcini, J., & Shea, J. A. (1992). Th e reproducibility of standards over groups and occasions. Applied Measurement in Education, 5(1), 63-72.
51. Norcini, J., Shea, J. A., & Kanya, D. T. (1988). Th e eff ect of various factors on standard setting. Journal of Educational Measurement, 25(1), 57-65.
52. O’Grady, W., Archibald, J., Aronoff, M., & Rees-Miller, J. (2009). Contemporary linguistics, 6th edition. Boston, MA: Bedford/St. Martin’s.
53. Pellegrino, J. W., Jones, L. R., & Mitchell, K. J. (1999). Grading the Nation’s Report Card: Evaluating NAEP and transforming the assessment of educational progress. Washington, DC: National Academy Press.
54. Perie, M. (2008). A guide to understanding and developing PLDs. Educational Measurement: Issues and Practice, 27(4), 15-29.
55. Perkins, M.R. (1979). Defi ning competencies. In W. P. Gorth & M. P. Perkins (Eds.), A study of minimum competency testing programs. Final program development resource document (pp. 25-50). Amherst, MA: National Evaluation Systems.
56. Plake, B. S., Huff, K., & Reshetar, R. (2010). Evidence-centered assessment design as a foundation for achievement-level descriptor development and for standard setting. Applied Measurement in Education, 23(4), 342-357.
57. Plake, B. S., & Impara, J. C. (2001). Ability of panelists to estimate item performance for a target group of candidates: An issue in judgmental standard setting. Educational Assessment, 7(2), 87-97.
58. Plake, B. S., Impara, J. C., & Potenza, M. T. (1994). Content specifi city of expert judgments in a standard-setting study. Journal of Educational Measurement, 13(4), 339-347.
59. Plous, S. (1993). Th e psychology of judgment and decision making. New York, NY: McGraw-Hill.
60. Rabinowitz, S. N., Roeber, E., Schroeder, C., & Sheinker, J. (2006). Creating aligned standards and assessment systems. Washington, DC: CCSSO.
61. Reckase, M. (2009). Multidimensional item response theory. New York, NY: Springer.
62. Ryan, J. (2003). An analysis of item mapping and test reporting strategies: Final Report. Retrieved February 17, 2011, from http://www.ed.sc.gov/agency/Accountability/Assessment/old/assessment/publications/serve020705.pdf
63. Schneider, M. C., Egan, K. L., Kim, D., & Brandstrom, A. (2008, March). Stability of achievement level descriptors across time and equating methods. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
64. Schneider, M. C., Egan, K. L., Siskind, T., Brailsford, A., & Jones, E. (2009, April). Concurrence of target student descriptors and mapped item demands in achievement levels across time. Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego, CA.
65. Schneider, M. C., Huff, K. L., Egan, K. L, Tully, M., & Ferrara, S. (2010, May). Aligning achievement level descriptors to mapped item demands to enhance valid interpretations of scale scores and inform item development. Paper presented at the annual meeting of the American Educational Research Association, Denver, CO.
66. Vinovskis, M. A. (1998). Overseeing the nation’s report card: Th e creation and evolution of the National Assessment Governing Board (NAGB). Washington, DC: United States Department of Education.
67. Zieky, M. J. (1995). A historical perspective on setting standards. Paper presented at the Joint Conference on Standard Setting for Large-Scale Assessments of the National Assessment Governing Board (NAGB) and the National Center for Education Statistics (NCES), Washington, DC.