Validity issues in standard-setting studies

Studies in Educational Evaluation

Volumn 35, Issue 2-3, 2009, Pages 95-101

  Pant, Hans A ^a   Rupp, André A ^b   Tiffin Richards, Simon P ^a   Köller, Olaf ^c  

^a FREIE UNIVERSITÄT BERLIN   (Germany)

^b UNIVERSITY OF MARYLAND   (United States)

^c HUMBOLDT UNIVERSITY   (Germany)

Author keywords

Accountability; Large scale assessment; Standard setting; Standards based testing; Validity

Indexed keywords

EID: 70949083655     PISSN: 0191491X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.stueduc.2009.10.008     Document Type: Review

References (56)

1. American Educational Research Association, American Psychological Association & National Council on Measurement in Education. Standards for educational and psychological testing (1999), American Educational Research Association, Washington, DC
2. Barron S., and Koretz D. Interpretation and use of the NAEP TSA results (final report) (1998), National Center for Education Statistics, Washington, DC
3. Bhola D.S., Impara J.C., and Buckendahl C.W. Aligning tests with states' content standards: Methods and issues. Educational Measurement: Issues and Practice 22 3 (2005) 21-29
4. Borsboom D., Mellenbergh G.J., and van Heerden J. The concept of validity. Psychological Review 111 (2004) 1061-1071
5. Buckendahl C.W., Smith R.W., Impara J.C., and Plake B.S. A comparison of Angoff and Bookmark standard setting methods. Journal of Educational Measurement 39 (2002) 253-263
6. Bullock C.D., and de Stefano L. A study of the utility of results from the 1992 Trial State Assessment (TSA) in reading for state-level administrators of assessment. Educational Evaluation and Policy Analysis 20 (1998) 47-51
7. Cizek G. Conjectures on the rise and call of standard-setting: An introduction to context and practice. In: Cizek G.J. (Ed). Setting performance standards: Concepts, methods, and perspectives (2001), Erlbaum, Mahwah, NJ 3-17
8. Cizek G.J., and Bunch M.B. Standard-setting. A guide to establishing and evaluating performance standards on tests (2007), Sage Publications Inc., California
9. Cronbach L.J., and Meehl P.E. Construct validity in psychological tests. Psychological Bulletin 52 (1955) 281-302
10. D'Agostino J.V., Welsh M.E., Cimetta A.D., Falco L.D., Smith S., VanWinkle W.H., et al. The rating and matching item-objective alignment methods. Applied Measurement in Education 21 (2008) 1-21
11. Dawber, T., Lewis, D. M., & Rogers, W. T. (2002, April). The cognitive experience of Bookmark standard-setting participants. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
12. Embretson S., and Gorin J. Improving construct validity with cognitive psychology principles. Journal of Educational Measurement 38 (2001) 343-368
13. Ferdous A.A., and Plake B.S. Understanding the factors that influence decisions of panelists in a standard-setting study. Applied Measurement in Education 18 (2005) 257-267
14. Glass G.V. Fertilizers, pills and magnetic strips: The fate of public education in America (2008), Charlotte, NC, Information Age Publishing, Inc.
15. Green D.R., Trimble C.S., and Lewis D.M. Interpreting the results of three different standard setting procedures. Educational Measurement: Issues and Practice 22 1 (2003) 22-32
16. Haertel E.H., and Lorie W.A. Validating standards-based test score interpretations. Measurement: Interdisciplinary Research & Perspective 2 (2004) 61-103
17. Hambleton R., and Meara K. Newspaper coverage of NAEP results 1990 to 1998. In: Bourque M.L., and Byrd S. (Eds). Student performance standards on the National Assessment of Educational Progress: Affirmations and improvements (2000), National Assessment Governing Board, Washington, DC 131-155
18. Hambleton R.C., and Pitoniak M.J. Setting performance standards. In: Brennan R. (Ed). Educational measurement. 4th ed. (2006), Praeger Publishers, Westport, CT 433-470
19. Hattie J., Jaeger R.M., and Bond L. Persistent methodological questions in educational testing. Review of Research in Education 24 (1999) 393-446
20. Kane M.T. Validating the performance standards associated with passing scores. Review of Educational Research 64 (1994) 425-461
21. Kane M.T. Current concerns in validity theory. Journal of Educational Measurement 38 (2001) 319-342
22. Kane M.T. Terminology, emphasis, and utility in validation. Educational Researcher 37 (2008) 76-82
23. Karantonis A., and Sireci S.G. The bookmark standard-setting-method: A literature review. Educational Measurement: Issues and Practice 25 1 (2006) 4-12
24. Kolen M. Data collection designs and linking procedures. In: Dorans N.J., Pommerich M., and Holland P.W. (Eds). Linking and aligning scores and scales (2007), Springer, New York 31-55
25. Leighton J.P., and Gierl M.J. Defining and evaluating models of cognition used in educational measurement to make inferences about examinees' thinking processes. Educational Measurement: Issues and Practice 26 2 (2007) 3-16
26. Lewis D.M., and Haug C.A. Aligning policy and methodology to achieve consistent across-grade performance standards. Applied Measurement in Education 18 (2005) 11-34
27. Lissitz R.W., and Samuelsen K. A suggested change in terminology and emphasis regarding validity and education. Educational Researcher 36 (2007) 437-448
28. Lissitz R.W., and Wei H. Consistency of standard-setting in an augmented state testing system. Educational Measurement: Issues and Practice 27 2 (2008) 46-55
29. McGinty D. Illuminating the "black box" of standard-setting: An exploratory qualitative study. Applied Measurement in Education 18 (2005) 269-287
30. Messick S. The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher 23 2 (1994) 13-23
31. Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist 50 (1995) 741-749
32. Messick S. Standards-based score interpretation: Establishing valid grounds for valid inferences. In: Crocker L., and Zieky M. (Eds). Proceedings of the joint conference on standard-setting for large-scale assessments, Vol. 2 (1995), U.S. Government Printing Office, Washington, DC 291-305
33. Mills C.N., Melican G.J., and Ahluwalia N.T. Defining minimal competence. Educational Measurement: Issues and Practice 10 2 (1991) 7-10
34. Mislevy R.J., Steinberg L.S., and Almond R.G. On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives 1 (2003) 3-62
35. Mitzel H.C., Lewis D.M., Patz R.J., and Green D.R. The Bookmark procedure: Psychological perspectives. In: Cizek G. (Ed). Setting performance standards: Concepts, methods and perspectives (2001), Erlbaum, Mahwah, NJ 249-281
36. Morgan D.L., and Michaelides M.P. Setting cut-scores for college placement (2005), The College Board, New York (College Board Research Report No. 2005-9)
37. Moss P.A., Girard B.J., and Haniford L.C. Validity in educational assessment. Review of Research in Education 30 (2006) 109-162
38. Norcini J.J. Setting standards on educational tests. Medical Education 37 (2003) 464-469
39. Norman R.J., and Buckendahl C.W. Determining sufficient measurement opportunities when using multiple cut-scores. Educational Measurement: Issues and Practice 27 1 (2008) 37-46
40. Pitoniak, M. J. (2003). Standard-setting methods for complex licensure examinations. Unpublished doctoral dissertation, University of Massachusetts, Amherst.
41. Plake B.S. Standard setters: Stand up and take a stand!. Educational Measurement: Issues and Practice 27 1 (2008) 3-9
42. Plake B.S., Melican G.L., and Mills C.N. Factors influencing intrajudge consistency during standard-setting. Educational Measurement: Issues and Practice 10 2 (1991) 15-16
43. Popham W.J. A tale of two test-specification strategies. Educational Measurement: Issues and Practice 11 2 (1992) 16-116
44. Porter A.C. Measuring the content of instruction: Uses in research and practice. Educational Researcher 31 7 (2002) 3-14
45. Raymond M.R., and Reid J.B. Who made thee a judge? Selecting and training participants for standard-setting. In: Cizek G.J. (Ed). Standard-setting: Concepts, methods, and perspectives (2001), Lawrence Erlbaum Associates, Mahwah, NJ 119-157
46. Resnick L.B., Rothman R., Slattery J.B., and Vranek J.L. Benchmarking and alignment of standards and testing. Educational Assessment 9 (2004) 1-27
47. Schafer W.D. Criteria for standard setting from the sponsor's perspective. Applied Measurement in Education 18 (2005) 61-81
48. Schulz E.M. Commentary: A response to Reckase's conceptual framework and examples for evaluating standard setting methods. Educational Measurement: Issues and Practice 25 3 (2006) 4-13
49. Sireci S.G. On validity theory and test validation. Educational Researcher 36 (2007) 477-481
50. Sireci, S. G. (2008, October). Packing and unpacking sources of validity evidence: History repeats itself again. Presented at the 9th annual Maryland assessment conference on the concept of validity: Revisions, new directions, and applications at the University of Maryland, College Park, MD.
51. Skaggs, G., & Tessema, A. (2001, April). Item disordinality with the bookmark standard-setting procedure. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
52. van der Linden W.J. A conceptual analysis of standard-setting in large-scale assessments. In: Crocker L., and Zieky M. (Eds). Proceedings of the joint conference on standard-setting for large-scale assessments, Vol. 2 (1995), National Assessment Governing Board & National Center for Education Statistics, Washington, DC 97-118
53. Wainer H., Hambleton R.K., and Meara K. Alternative displays for communicating NAEP results: A redesign and validity study. Journal of Educational Measurement 36 (1999) 301-335
54. Wang N. Use of the Rasch IRT model in standard setting: An item mapping method. Journal of Educational Measurement 40 (2003) 231-253
55. Webb N.L. Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education 20 (2007) 7-25
56. Zieky M., and Perie M. A primer on setting cut scores on tests of educational achievement (2006), Educational Testing Service, Princeton, NJ