Assessing the accuracy and consistency of language proficiency classification under competing measurement models

Language Testing

Volumn 27, Issue 1, 2010, Pages 119-140

  Zhang, Bo ^a  

^a UNIVERSITY OF WISCONSIN MILWAUKEE   (United States)

Author keywords

Item response theory; Language proficiency; Language testing; Proficiency classification; Testlet

Indexed keywords

EID: 75749104988     PISSN: 02655322     EISSN: 14770946     Source Type: Journal    
DOI: 10.1177/0265532209347363     Document Type: Article

References (49)

1. Allen MJ and Yen WM (1979). Introduction to Measurement Theory. Monterey, CA: Brooks/Cole.
2. Adams RJ, Griffin PE and Martin L. (1987). A latent trait method for measuring a dimension in second language proficiency. Language Testing, 4(1), 9-28.
3. Alderson JC (1979). The cloze procedure and proficiency in English as a foreign language. TESOL Quarterly, 13(2), 219-223.
4. Bachman LF (1982). The trait structure of cloze test scores. TESOL Quarterly, 16(1), 61-70.
5. Bachman LF (1991). What does language testing have to offer? TESOL Quarterly, 25(4), 671-704.
6. Bachman LF and Palmer AS (1996). Language testing in practice: Designing and developing useful language tests. Oxford, UK: Oxford University Press.
7. Birnbaum A. (1968). Some latent trait models and their use in inferring an examinee's ability. In M. Lord and M. R. Novick (Eds), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.
8. Bradlow ET, Wainer H. and Wang X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64(2), 153-168.
9. Brennan RL (2004). Manual for BB-CLASS: A computer program that uses the beta-binomial model for classification consistency and accuracy, Version 1.1 (CASMA Research Report No. 9). Iowa City, IA: University of Iowa.
10. Canale M. and Swain M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1-47.
11. Cizek G (Ed). (2001). Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum.
12. Cronbach LJ (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.
13. English Language Institute, University of Michigan. (2006). Examination for the Certificate of Proficiency in English 2004-05 annual report. Ann Arbor, MI: English Language Institute, University of Michigan.
14. Guo FM (2006). Expected classification accuracy using the latent distribution. Practical Assessment, Research and Evaluation, 11(6).
15. Keller LA, Swaminathat H. and Sireci SG (2003). Evaluating scoring procedures for context dependent item sets1. Applied Measurement in Education, 16(3), 207-222.
16. Hambleton R. and Novick M. (1973). Toward an integration of theory and method for criterion-referenced tests. Journal of Educational Measurement, 10(3), 159-170.
17. Hambleton RK and Swaminathan H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer.
18. Hanson BA and Brennan RL (1990). An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement, 27(4), 345-359.
19. Harwell MR and Janosky JE (1991). An empirical study of the effects of small datasets and varying prior variances on item parameter estimation in BILOG. Applied Psychological Measurement, 15(3), 279-291.
20. Huynh H. (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13(4), 253-264.
21. Kolen MJ, Hanson BA and Brennan RL (1992). Conditional standard errors of measurement for scale scores. Journal of Educational Measurement, 29, 285-307.
22. Lee W., Hanson BA and Brennan RL (2002). Estimating consistency and accuracy indices for multiple classifications. Applied Psychological Measurement, 26, 412-432.
23. Livingston SA and Lewis C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32(2), 179-197.
24. Lord FM (1965). A strong true-score theory with applications. Psychometrika, 30(3), 239-270.
25. Lord FM (1980). Applications of item response theory to practical testing problems. Mahwah, NJ: Lawrence Erlbaum.
26. Lynch BK, Davidson F. and Henning G. (1988). Person dimensionality in language test validation. Language Testing, 5(2), 206-219.
27. McNamara TF (1990). Item response theory and the validation of an ESP test for health professionals. Language Testing, 7(1), 52-76.
28. Nunnally JC and Bernstein IH (1994). Psychometric theory. New York: McGraw-Hill.
29. Oller JW Jr (1973). Cloze tests of second language proficiency and what they measure. Language Learning, 23, 105-118.
30. Oller JW Jr (1979). Language tests at school: A pragmatic approach. London: Longman.
31. Peterson NS, Kolen MJ and Hoover HD (1989). Scaling, norming, and equating. In Linn, R. L. (Ed) Educational Measurement (3rd Edn.) (pp. 221-262). New York: American Council on Education and Macmillan.
32. Rosenbaum PR (1988). Item bundles. Psychometrika, 53(3), 349-359.
33. Rudner LM (2001). Computing the expected proportions of misclassified examinees. Practical Assessment, Research and Evaluation, 7(14).
34. Rudner LM (2003). The Classification Accuracy of Measurement Decision Theory. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, 23-25 April 2003.
35. Rudner LM (2005). Expected classification accuracy. Practical Assessment Research and Evaluation, 10(13). Available online: http://pareonline.net/getvn.asp?v=10andn=13.
36. Samejima F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, Monograph Supplement, No. 17.
37. Saito Y. (2003). Investigating the construct validity of the cloze section in the Examination for the Certificate of Proficiency in English. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 2, 39-82.
38. Sireci SG, Thissen D. and Wainer H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28(3), 237-247.
39. Spearman C. (1910). Correlation calculated with faulty data. British Journal of Psychology, 3, 271-295.
40. Spiegelhalter D., Thomas A. and Best N. (2003). WinBUGS version 1.4 [computer program]. Robin-son Way, Cambridge CB2 2SR, UK: MRC Biostatistics Unit, Institute of Public Health.
41. Subkoviak M. (1976). Estimating reliability from a single administration of a criterion-referenced test. Journal of Educational Measurement, 13(4), 265-275.
42. Thissen D. (1991). MULTILOG 6.3 [Computer program]. Mooresville, IN: Scientific Software.
43. Thissen D., Steinberg L. and Mooney JA (1989). Trace lines for testlets: A use of multiplecategorical-response models. Journal of Educational Measurement, 26(3), 247-260.
44. Wainer H. and Kiely GL (1987). Item clusters and computerized-adaptive testing: A case for testlets. Journal of Educational Measurement, 24(3), 185-201.
45. Wainer H., Wang X., Skorupski WPet al.(2005). A Bayesian method for evaluating passing scores: the PPoP curve. Journal of Educational Measurement, 2(3), 271-281.
46. Wan L., Brennan RL and Lee W. (2007). Estimating classification consistency for complex assessments. (CASMA Research Report No. 22). Iowa City, IA: University of Iowa.
47. Wang X., Bradlow ET and Wainer H. (2002). A general Bayesian model for testlets: Theory and application. Applied Psychological Measurement, 26(1), 109-128.
48. Wang X., Bradlow ET and Wainer H. (2004). User's guide for SCORIGHT (version 3.0): A computer program for scoring tests built of testlets including a module for covariate analysis. Research Report 04-49. Princeton, NJ: Educational Testing Services.
49. Yen WM (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30(3), 187-213.