A qualitative investigation of panelists' experiences of standard setting using two variations of the bookmark method

Applied Measurement in Education

Volumn 22, Issue 3, 2009, Pages 207-228

  Hein, Serge F ^a   Skaggs, Gary E ^a  

^a VIRGINIA POLYTECHNIC INSTITUTE AND STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 74949101877     PISSN: 08957347     EISSN: None     Source Type: Journal    
DOI: 10.1080/08957340902983978     Document Type: Article

References (39)

1. Becker, C. S. (1986). Interviewing in human science research. Methods, 1, 101-124.
2. Berg, B. L. (2004). Qualitative research methods for the social sciences (5th ed.). Boston: Allyn & Bacon.
3. Bogdan, R. C., & Biklen, S. K. (2002). Qualitative research in education: An introduction to theory and methods (4th ed.). Boston: Allyn & Bacon.
4. Brandon, P. R. (2004). Conclusions about frequently studied modified Angoff standard-setting topics. Applied Measurement in Education, 17, 59-88.
5. Buckendahl, C. W. (2005). Qualitative inquiries of participants' experiences with standard setting. Applied Measurement in Education, 18, 219-221.
6. Buckendahl, C. W., Smith, R. W., Impara, J. C., & Plake, B. S. (2002). A comparison of Angoff and Bookmark standard setting methods. Journal of Educational Measurement, 39, 253-263.
7. Dawber, T., Lewis, D. M., & Rogers, W. T. (2002). The cognitive experience of bookmark standard setting participants. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.
8. Egan, K., & Green, D. R. (2003). Influences on judges' decisions. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
9. Ferdous, A. A., & Plake, B. S. (2005). Understanding the factors that influence decisions of panelists in a standard setting-study. Applied Measurement in Education, 18, 257-267.
10. Fern, E. F. (2001). Advanced focus group research. Thousand Oaks, CA: Sage.
11. Fitzpatrick, A. (1989). Social influences in standard setting: The effects of social interaction on group judgments. Review of Educational Research, 59, 315-328.
12. Giraud, G., & Impara, J. C. (2005). Making the cut: The cut score setting process in a public school district. Applied Measurement in Education, 18, 289-312.
13. Giraud, G., Impara, J. C., & Plake, B. S. (2005). Teachers' conceptions of the target examinee in Angoff standard setting. Applied Measurement in Education, 18, 223-232.
14. Greenbaum, T. L. (1998). The handbook for focus group research (2nd ed.). Thousand Oaks, CA: Sage.
15. Hertz, N. R., & Chinn, R. N. (2002). The role of deliberation style in standard setting for licensing and certification examinations. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA.
16. Impara, J. C., & Plake, B. S. (1998). Teachers' ability to estimate item difficulty: A test of the assumptions of the Angoff standard setting method. Journal of Educational Measurement, 35, 69-81.
17. Karantonis, A., & Sireci, S. G. (2006). The bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4-12.
18. Krathwohl, D. R. (1998). Methods of educational and social science research: An integrated approach (2nd ed.). New York: Longman.
19. Lewis, D. M. (2000). State implementations of the bookmark standard setting procedure. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA.
20. Lewis, D. M., & Green, D. R. (1997). The validity of performance level descriptors. Paper presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
21. Lewis, D. M., Mitzel, H. C., & Green, D. R. (1996). Standard setting: A bookmark approach. Paper presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
22. Loomis, S. C. (2000). Feedback in the NAEP achievement levels setting process. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA.
23. McGinty, D. (2005). Illuminating the "black box" of standard setting: An exploratory qualitative study. Applied Measurement in Education, 18, 269-287.
24. Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2001). The bookmark procedure: Psychological perspectives. In C. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 249-281). Mahwah, NJ: Lawrence Erlbaum.
25. Mehrens, W. A. (1995). Methodological issues in standard setting for educational exams. In Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments, Volume II (pp. 221-263). Washington DC: National Assessment Governing Board and National Center for Education Statistics.
26. Miles, M., & Huberman, A. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage.
27. No Child Left Behind Act (NCLB) of 2001, 20 U.S.C. §6311 et seq. (2002).
28. Norcini, J. J., & Shea, J. A. (1997). The credibility and comparability of standards. Applied Measurement in Education, 10, 39-59.
29. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.
30. Plake, B. S., & Impara, J. C. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34, 353-366.
31. Plake, B. S., Impara, J. C., Hertzog, M., Giraud, G., & Spies, R. (1997). Utility of a concept-focusing strategy on judgmental standard setting results. Paper presented at the Annual Meeting of the Midwestern Educational Research Association, Chicago, IL.
32. Plake, B., Melican, G., & Mills, C. (1991). Factors influencing intrajudge consistency during standard setting. Educational Measurement: Issues and Practice, 10(2), 15-26.
33. Reid, J. B. (1991). Training judges to generate standard-setting data. Educational Measurement: Issues and Practice, 10(2), 11-14.
34. Shepard, L. (1993). Evaluating test validity. Review of Research in Education, 19, 405-450.
35. Shepard, L. A. (1995). Implications for standard setting of the National Academy of Education evaluation of the National Assessment of Educational Progress achievement levels. In Proceedings of the Joint Conference on Standard Setting for Large-Scale Assessments, Volume II (pp. 143-160).Washington DC: National Assessment Governing Board and National Center for Education Statistics.
36. Skaggs, G., Hein, S. F., & Awuor, R. (2007). Setting passing scores on passage-based tests: A comparison of traditional and single-passage bookmark methods. Applied Measurement in Education, 20, 405-426.
37. Skaggs, G., & Tessema, A. (2001). Item disordinality with the bookmark standard setting procedure.Paper presented at the Annual Meeting of the National Council on Measurement in Education, Seattle, WA.
38. Skorupski, W. P., & Hambleton, R. K. (2005). What are panelists thinking when they participate in standard-setting studies? Applied Measurement in Education, 18, 233-256.
39. Ziecky, M. J. (2001). So much has changed: How the setting of cutscores has evolved since the 1980s. In C. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 19-51). Mahwah, NJ: Lawrence Erlbaum.