Critical factors to consider in evaluating standard-setting studies to map language test scores to frameworks of language proficiency

Language Assessment Quarterly

Volumn 11, Issue 3, 2014, Pages 233-249

  Tannenbaum, Richard J ^a   Cho, Yeonsuk ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84906890725     PISSN: 15434303     EISSN: 15434311     Source Type: Journal    
DOI: 10.1080/15434303.2013.869815     Document Type: Article

References (61)

1. American Council on the Teaching of Foreign Languages. (2012). ACTFL proficiency guidelines. Alexandria, VA: Author.
2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing.Washington, DC: American Psychological Association.
3. Bachman, L. F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1-34.
4. Bejar, I. I., Braun, H. I., & Tannenbaum R. J. (2007). A prospective, progressive, and predictive approach to standard setting. In R. W. Lissitz (Ed.), Assessing and modeling cognitive development in school (pp. 1-30). Maple Grove, MN: JAM Press.
5. Bhola, D., Impara, J. C., & Buckendahl, C. W. (2003). Aligning tests with states' content standards: Methods and Issues. Educational Measurement: Issues and Practice, 22(3), 21-29.
6. Brennan, R. L. (2002, October). Estimated standard error of a mean when there are only two observations (Center for Advanced Studies in Measurement and Assessment Technical Note No. 1). Iowa City: University of Iowa, CASMA.
7. Centre for Canadian Language Benchmarks. (2012). Canadian Language Benchmarks: English as a Second Language for Adults. Ottawa, Canada: Ministry of Citizenship and Immigration.
8. Cizek, G.J. (1993). Reconsidering standards and criteria. Journal of Educational Measurement, 30(2), 93-106.
9. Cizek, G. J. (2012a). The forms and functions of evaluations in the standard setting process. In G. J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 165-178). New York, NY: Routledge.
10. Cizek, G. J. (Ed.). (2012c). Setting performance standards: Foundations, methods, and innovations (2nd ed.). New York, NY: Routledge.
11. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.
12. Cizek, G. J., Bunch, M. B., & Koons, H. (2004). Setting performance standards: Contemporary methods. Educational Measurement: Issues and Practice, 23(4), 31-50.
13. Clauser, B. E., Mee, J., Baldwin, S. G., Margolis, M. J., & Dillon, G. F. (2009). Judge's use of examinee performance data in an Angoff standard-setting exercise for a medical licensing examination: An experimental study. Journal of Educational Measurement, 46(4), 390-407.
14. Cohen, A. S., Kane, M. T., & Crooks, T. J. (1999). A generalized examinee-centered method for setting standards on achievement tests. Applied Measurement in Education, 12(4), 343-366.
15. Cohen, R. J., Swerdlik, M. E., & Smith, D. K. (1992). Psychological testing and assessment: An introduction to tests and measurement (2nd ed.). Mountain View, CA: Mayfield.
16. Council of Europe.(2001). Common European Framework of Reference for Languages: Learning, Teaching, Assessment). Cambridge, UK: Cambridge University Press.
17. Council of Europe. (2009). Relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment. Retrieved from http://www.coe.int/t/dg4/linguistic/manuel1-en.asp#P75-5622 asp#P75-5622
18. Davis-Becker, S. L., & Buckendahl, C. W. (2013). A proposed framework for evaluating alignment studies. Educational Measurement: Issues and Practice, 32(1), 23-33.
19. Egan, K. L., Schneider, M. C., & Ferrara, S. (2012). Performance level descriptors: History, practice, and a proposed framework. In G. J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 79-106). New York, NY: Routledge.
20. Figueras, N., & Noijons, J. (Eds.). (2009). Linking to the CEFR levels: Research perspectives. Arnhem, the Netherlands: CITO.
21. Hambleton, R. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 89-116). Mahwah, NJ: Erlbaum.
22. Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 433-470). Westport, CT: American Council on Education/Praeger.
23. Hambleton, R. K., Pitoniak, M. J., & Copella, J. M. (2012). Essential steps in setting performance standards on educational tests and strategies for assessing the reliability of results. In G. J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 47-76). New York, NY: Routledge.
24. Hambleton, R. K., & Plake, B. S. (1995). Using an extended Angoff procedure to set standards on complex performance assessments. Applied Measurement in Education, 8(1), 41-55.
25. Huff, K., Steinberg, L., & Matts, T. (2010). The promises and challenges of implementing evidence-centered design in large-scale assessment. Applied Measurement in Education, 23(4), 310-324.
26. Impara, J. C., & Plake, B. S. (1997). Standard setting: An alternative approach. Journal of Educational Measurement, 34(4), 353-366.
27. Interagency Language Roundtable. (1985). Interagency language roundtable skill descriptions. Retrieved from http:/www.govilr.org
28. Kaftandjieva, F. (2010). Methods for setting cut scores in criterion-referenced achievement tests: A comparative analysis of six methods with an application to tests of reading in EFL. Arnhem, the Netherlands: CITO.
29. Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425-461.
30. Kane, M. T. (2001). So much remains the same: Conception and status of validation in setting standards. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 53-88). Mahwah, NJ: Erlbaum.
31. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: American Council on Education/Praeger.
32. Kane, M. (2012). Validating score interpretations and uses. Language Testing, 29(1), 3-17.
33. Kantarcioglu, E., & Papageorgiou, S. (2011). Benchmarking and standards in language tests. In B. O'Sullivan (Ed.), Language testing: Theories and practices (pp. 94-110). Basingstoke, UK: Palgrave MacMillan.
34. Karantonis, A., & Sireci, S. G. (2006). The Bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4-12.
35. Kingston, N. M., & Tiemann, G. C. (2012). Setting standards on complex assessments: The Body of Work method. In G. J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 201-223). New York, NY: Routledge.
36. Lewis, D. M., Mitzel, H. C., Mercado, R. L., & Schultz, E. M. (2012). The Bookmark standard setting procedure. In G. J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 225-253). New York, NY: Routledge.
37. Loomis, S. C., & Bourque, M. L. (2001). From tradition to innovation: Standard setting on the National Assessment of Educational Progress. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 175-217). Mahwah, NJ: Erlbaum.
38. Martyniuk, W. (Ed.). (2010). Aligning tests with the CEFR: Reflections on using the Council of Europe's draft manual. Cambridge, UK: Cambridge University Press.
39. Morgan, D. L., & Perie, M. (2004, June). Setting standards in education: Choosing the best method for your assessment and population. Paper presented at the annual meeting of the International Association for Educational Assessment, Philadelphia, PA.
40. Nichols, P., Twing, J., Mueller, C. D., & O'Malley, K. (2010). Standard-setting methods as measurement processes. Educational Measurement: Issues and Practice, 29(1), 14-24.
41. North Atlantic Treaty Organization. (2010). Standardization Agreement (STANAG 6001, 4th ed.). Brussels, Belgium: NATO Standardization Agency.
42. Papageorgiou, S. (2010). Setting cut scores on the Common European Framework of Reference for the Michigan English Test. Ann Arbor, MI: University of Michigan.
43. Perie, M. (2008). A guide to understanding and developing performance-level descriptors. Educational Measurement: Issues and Practice, 27(4), 15-29.
44. Plake, B. S. (2008). Standard setters: Stand up and take a stand. Educational Measurement: Issues and Practice, 27(1), 3-9.
45. Plake, B. S., & Cizek, G. J. (2012). Variations on a theme: The Modified Angoff, Extended Angoff, and Yes/No standard setting methods. In G. J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 181-199). New York, NY: Routledge.
46. Plake, B. S., Huff, K., & Reshetar, R. (2010). Evidence-centered assessment design as a foundation for achievement-level descriptor development and for standard setting. Applied Measurement in Education, 23(4), 342-357.
47. Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3-14.
48. Raymond, M. R., & Reid, J. R. (2001). Who made thee a judge Selecting and training participants for standard setting. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 119-157). Mahwah, NJ: Erlbaum.
49. Reckase, M. D. (2001). Innovative methods for helping standard-setting participants to perform their task: The role of feedback regarding consistency, accuracy, and impact. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 159-173). Mahwah, NJ: Erlbaum.
50. Reckase, M. D. (2006). A conceptual framework for a psychometric theory for standard setting with examples of its use for evaluating the functioning of two standard setting methods. Educational Measurement: Issues and Practice, 25(2), 4-18.
51. Tannenbaum, R. J. (2011). Standard setting. In J.W. Collins & N. P. O'Brien (Eds.), Greenwood Dictionary of Education (2nd ed., p. 441). Santa Barbara, CA: ABC-CLIO
52. Tannenbaum, R. J. (2012). A multistate approach to setting standards: An application to teacher licensure tests. CLEAR Exam Review, 23(1), 18-24.
53. Tannenbaum, R. J., & Baron, P. A. (2010). Mapping TOEIC® test scores to the STANAG 6001 language proficiency levels (RM 10-11). Princeton, NJ: Educational Testing Service.
54. Tannenbaum, R. J., & Baron, P. A. (2011). Mapping TOEFL® ITP Scores Onto the Common European Framework of Reference (RM 11-33). Princeton, NJ: Educational Testing Service.
55. Tannenbaum, R. J., & Katz, I. R. (2013). Standard setting. In K. F. Geisinger (Ed.), APA handbook of testing and assessment in psychology: Vol. 3. Testing and assessment in school psychology and education (pp. 455-477). Washington, DC: American Psychological Association.
56. Tannenbaum, R. J., &Wylie E. C. (2005). Mapping English language proficiency test scores onto the Common European Framework (RR-05-18). Princeton, NJ: Educational Testing Service.
57. Tannenbaum, R. J., & Wylie, E. C. (2008). Linking English language test scores onto the Common European Framework of Reference: An application of standard setting methodology (TOEFL iBT Series Rep. No. TOEFLibt-06, RR-08-34). Princeton, NJ: Educational Testing Service.
58. Webb, N. L. (1999). Alignment of science and mathematics standards and assessments in four states (Council of Chief State School Officers and National Institute for Science Education). Madison: Wisconsin Center for Education Research, University of Wisconsin.
59. Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7-25.
60. Zieky, M. J. (2001). So much has changed: How the setting of cutscores has evolved since the 1980s. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 19-51). Mahwah, NJ: Erlbaum.
61. Zieky, M. J., Perie, M, & Livingston, S. A. (2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.