Illuminating the "black box" of standard setting: An exploratory qualitative study

Applied Measurement in Education

Volumn 18, Issue 3, 2005, Pages 269-287

  McGinty, Dixie ^a  

^a WESTERN CAROLINA UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 23844518737     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1207/s15324818ame1803_5     Document Type: Article

References (47)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
2. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
3. Beck, M. D. (2003, April). Standard setting: If it is science, it's sociology & linguistics, not psychometrics. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
4. Berk, R. A. (1995). Standard setting - The next generation. In L. Crocker & M. Zieky (Eds.), Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 161-181). Washington, DC: U.S. Government Printing Office.
5. Berk, R. A. (1996). Standard setting: The next generation (Where few psychometricians have gone before!). Applied Measurement in Education, 9, 215-235.
6. Bond, L. (1995). Ensuring fairness in the setting of performance standards. In L. Crocker & M. Zieky (Eds.), Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 311-321). Washington, DC: U.S. Government Printing Office.
7. Brennan, R. L. (1995). Standard setting from the perspective of generalizability theory. In L. Crocker & M. Zieky (Eds.), Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 269-287). Washington, DC: U.S. Government Printing Office.
8. Cizek, G. J. (1995, April). Standard setting as psychometric due process: Going a little further down an uncertain road. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco.
9. Cohen, A. S., Kane, M. T., & Crooks, T. J. (1999). A generalizable examinee-centered method for setting standards on achievement tests. Applied Measurement in Education, 12, 367-381.
10. Dawber, T., Lewis, D. M., & Rogers, W. T. (2002, April). The cognitive experience of bookmark standard setting participants. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
11. DeMauro, G. E. (2003, April). Developing a theory of performance: A two-stage structure for the psychology of standard setting. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
12. Egan, K., & Green, D. R. (2003, April). Influences on judges' decisions. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
13. Engelhard, G. E., & Anderson, D. W. (1998). A binomial trials model for examining the ratings of standard setting panelists. Applied Measurement in Education, 11, 209-230.
14. Fitzpatrick, A. (1989). Social influences in standard setting: The effects of social interaction on group judgments. Review of Educational Research, 59, 315-328.
15. Geisinger, K. F. (1991). Using standard setting data to establish cutoff scores. Educational Measurement: Issues and Practice, 70(2), 17-22, 25-26.
16. Glass, G. V. (1978). Standards and criteria. Journal of Educational Measurement, 15, 237-261.
17. Hambleton, R. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 89-116). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
18. Impara, J. C., & Plake, B. S. (1998). Teachers' ability to estimate item difficulty: A test of the assumptions in the Angoff standard setting method. Journal of Educational Measurement, 35, 69-81.
19. Jaeger, R. M. (1995). On the cognitive construction of standard setting judgments: The case of configural scoring. In L. Crocker & M. Zieky (Eds.), Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 57-70). Washington, DC: U.S. Government Printing Office.
20. Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527-535.
21. Kane, M. T. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.
22. Kane, M. T. (2001). So much remains the same: Conception and status of validation in setting standards. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
23. Lewis, D. M., Mitzel, H. C., Green, D. R., & Patz, R. J. (1999). The bookmark standard setting procedure. Monterey, CA: McGraw-Hill.
24. Linn, R. L. (1998). Validating inferences from NAEP achievement-level reporting. Applied Measurement in Education, 11, 23-47.
25. Loomis, S. C. (2000, April). Feedback in the NAEP achievement levels setting process. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
26. Madaus, G. F. (1986). Measurement specialists: Testing the faith - A reply to Mehrens. Educational Measurement: Issues and Practice, 5(4), 11-14.
27. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: American Council on Education/Macmillan.
28. Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
29. Messick, S. (1995). Standards-based score interpretation: Establishing valid grounds for valid inferences. In L. Crocker & M. Zieky (Eds.), Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 291-305). Washington, DC: U.S. Government Printing Office.
30. Mills, C. N., Melican, G. J., & Ahluwalia, N. T. (1991). Defining minimal competence. Educational Measurement: Issues and Practice, 10(2), 7-10, 14.
31. Norcini, J. J., & Shea, J. A. (1997). The credibility and comparability of standards. Applied Measurement in Education, 10, 39-59.
32. Plake, B. S. (1998). Setting performance standards for professional licensure and certification. Applied Measurement in Education, 11, 65-80.
33. Plake, B. S., & Impara, J. C. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353-366.
34. Plake, B. S., Impara, J. C., & Irwin, P. M. (2000). Consistency of Angoff-based predictions of item performance: Evidence of technical quality of results from the Angoff standard setting method. Journal of Educational Measurement, 37, 347-355.
35. Plake, B. S., Melican, G. J., & Mills, C. N. (1991). Factors influencing intrajudge consistency during standard setting. Educational Measurement: Issues and Practice, 10(2), 15-16, 22, 25-26.
36. Popham, W. J. (1995, April). Standard setting: What really sways the setters? Presentation at the annual meeting of the American Educational Research Association, San Francisco.
37. Reckase, M. D. (1998). Converting boundaries between National Assessment Governing Board performance categories to points on the NAEP score scale: The 1996 science NAEP process. Applied Measurement in Education, 11, 9-21.
38. Reid, J. B. (1991). Training judges to generate standard setting data. Educational Measurement: Issues and Practice, 10(2), 11-14.
39. Shepard, L. (1980). Standard setting issues and methods. Applied Psychological Measurement, 4, 447-467.
40. Shepard, L. (1993). Evaluating test validity. Review of Research in Education, 19, 405-450.
41. Shepard, L. (1995). Implications for standard setting of the National Academy of Education Evaluation of the National Assessment of Educational Progress achievement levels. In L. Crocker & M. Zieky (Eds.), Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 143-160). Washington, DC: U.S. Government Printing Office.
42. Sireci, S.G., Robin, F., & Patelis, T. (1997, March). Using cluster analysis to facilitate the standard setting process. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
43. Skorupski, W., & Hambleton, R. K. (2003, April). What are panelists really thinking when they set performance standards? Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago.
44. van der Linden, W. J. (1982). A latent trait method for determining the intrajudge inconsistency in the Angoff and Nedelsky techniques of setting standards. Journal of Educational Measurement, 19, 295-308.
45. van der Linden, W. J. (1995). A conceptual analysis of standard setting in large-scale assessments. In L. Crocker & M. Zieky (Eds.), Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 97-118). Washington, DC: U.S. Government Printing Office.
46. Zieky, M. J. (1995). A historical perspective on setting standards. In L. Crocker & M. Zieky (Eds.), Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments (Vol. 2, pp. 1-37). Washington, DC: U.S. Government Printing Office.
47. Zieky, M. J. (2001). So much has changed: How the setting of cut scores has evolved since the 1980s. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 19-51). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.