Using multilevel modeling in language assessment research: A conceptual Introduction

Language Assessment Quarterly

Volumn 10, Issue 3, 2013, Pages 241-273

  Barkaoui, Khaled ^a  

^a YORK UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84883076594     PISSN: 15434303     EISSN: 15434311     Source Type: Journal    
DOI: 10.1080/15434303.2013.769546     Document Type: Article

References (67)

1. Aitkin, M., Anderson, D., and, Hinde, J. 1981. Statistical modeling of data on teaching styles (with discussion). Journal of the Royal Statistical Society, A, 144: 148-161.
2. Bachman, L. F. 1997. " Generalizability theory. In C. Clapham (Vol. Ed.) & D. Corson (Series Ed.) ". In Language assessment, Vol. 7: Encyclopaedia of language and education, 255-262. Dordrecht, the Netherlands: Kluwer.
3. Barkaoui, K. 2010. Explaining ESL essay holistic scores: A multilevel modeling approach. Language Testing, 27 (4): 515-535.
4. Barkaoui, K. 2011. Effects of marking method and rater experience on ESL essay scores. Assessment in Education: Principles, Policy & Practice, 18 (3): 277-291.
5. Bickel, R. 2007. Multilevel analysis for applied research: It's just regression, New York, NY: Guilford.
6. Bond, T. G., and, Fox, C. M. 2001. Applying the Rasch model: Fundamental measurement in the human sciences, Mahwah, NJ: Erlbaum.
7. Brennan, R. L. 2001. Generalizability theory, New York, NY: Springer.
8. Cheong, Y. F. 2006. Analysis of school context effects on differential item functioning using hierarchical generalized linear models. International Journal of Testing, 6: 57-79.
9. Chiang, S. Y. 1999. Assessing grammatical and textual features in L2 writing samples: The case of French as a foreign language. The Modern Language Journal, 83: 219-232.
10. Daly, J. A., and, Dickson-Markman, F. 1982. Contrast effects in evaluating essays. Journal of Educational Measurement, 19: 309-316.
11. Davison, M. L., Kwak, N., Seo, Y. S., and, Choi, J. 2002. Using hierarchical linear models to examine moderator effects: Person-by-organization interactions. Organizational Research Methods, 5: 231-254.
12. Deadrick, D. L., Bennett, N., and, Russell, C. J. 1997. Using hierarchical linear modeling to examine dynamic performance criteria over time. Journal of Management, 23: 745-757.
13. de Leeuw, J., and, Kreft, I. G. G. 2001. " Software for multilevel analysis ". In Multilevel modelling of health statistics, Edited by: Leyland, A. H. and Goldstein, H. 189-204. West Sussex, UK: Wiley.
14. Elder, C. 1993. How do subject specialists construe classroom language proficiency?. Language Testing, 10: 235-254.
15. Enders, C. K., and, Tofighi, D. 2007. Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12: 121-138.
16. Feast, V. 2002. The impact of IELTS scores on performance at university. International Education Journal, 3: 70-85.
17. Feedle, R., and, Kostin, I. 1999. Does the text matter in a multiple-choice test of comprehension? The case for the construct validity of TOEFL's minitalks. Language Testing, 16: 2-32.
18. Field, A. 2009. Discovering statistics using SPSS, 3rd, Los Angeles, CA: Sage.
19. Fulcher, G. 2003. Testing second language speaking, New York, NY: Longma.
20. Goldstein, H. 1991. Nonlinear multilevel models with an application to discrete response data. Biometrika, 78: 45-51.
21. Goldstein, H. 2003. Multilevel statistical models, 3rd, London, UK: Edward Arnold.
22. Hales, L. W., and, Tokar, E. 1975. The effect of the quality of preceding responses on the grades assigned to subsequent responses to an essay question. Journal of Educational Measurement, 12: 115-117.
23. Heck, R. H., and, Thomas, S. L. 2000. An introduction to multilevel modeling techniques, Mahwah, NJ: Lawrence Erlbaum.
24. Hofmann, D. A. 1997. An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23: 723-744.
25. Hox, J. J. (1995). Applied multilevel analysis. Amsterdam, the Netherlands: T-T Publikaties. http://www.fss.uu.nl/ms/jh/publist/amaboek.pdf (http://www.fss.uu.nl/ms/jh/publist/amaboek.pdf)
26. Hox, J. J. 2002. Multilevel analysis: Techniques and applications, Mahwah, NJ: Erlbaum.
27. Hughes, D. C., and, Keeling, B. 1984. The use of model essays to reduce context effects in essay scoring. Journal of Educational Measurement, 21: 277-281.
28. Hughes, D. C., Keeling, B., and, Tuck, B. F. 1980a. Essay marking and the context problem. Educational Research, 22: 147-148.
29. Hughes, D. C., Keeling, B., and, Tuck, B. F. 1980b. The influence of context position and scoring method on essay scoring. of Educational Measurement, 17: 131-135.
30. In'nami, Y., and, Koizumi, R. 2011. Structural equation modeling in language testing and learning research: A review. Language Assessment Quarterly, 8: 250-276.
31. Kamata, A. 2001. Item analysis by the hierarchical generalized linear model. Journal of Educational Measurement, 38: 79-93.
32. Kenny, D. A., Kashy, D. A., and, Cook, W. L. 2006. Dyadic data analysis, New York, NY: Guilford Press.
33. Kreft, I., and, de Leeuw, J. 1998. Introducing multilevel modeling, Thousand Oaks, CA: Sage.
34. Kreft, I. G. G. (1996). Are multilevel techniques necessary? An overview, including simulation studies. http://www.calsttela.edu/faculty/ikreft/ quarterly/quarterly.html (http://www.calsttela.edu/faculty/ikreft/quarterly/ quarterly.html)
35. Kreft, I. G. G., de Leeuw, J., and, van der Leeden, R. 1994. Review of five multilevel analysis programs: BMDP-5V, GENMOD, HLM, ML3, VARCL. American Statistician, 48: 324-335.
36. Kunnan, A. J. 1998. An introduction to structural equation modeling for language assessment research. Language Testing, 15: 295-332.
37. Littell, R. C., Milliken, G. A., Stroup, W. W., and, Wolfinger, R. D. 1996. SAS system for mixed models, Cary, NC: SAS Institute.
38. Luke, D. 2004. Multilevel modeling, Thousand Oaks, CA: Sage.
39. Mass, C. J., and, Hox, J. J. 2005. Sufficient sample size for multilevel modeling. Methodology, 1: 86-92.
40. McLeod, A. 2001. " Multivariate multilevel models ". In Multilevel modeling of health statistics, Edited by: Leyland, A. H. and Goldstein, H. 59-74. Chichester, England: Wiley.
41. McNamara, T. F. 1990. Item response theory and the validation of an ESP test for health professionals. Language Testing, 7: 52-76.
42. McNamara, T. 1996. Measuring second language performance, London, UK: Longman.
43. O'Dwyer, L. M. 2005. Examining the variability of mathematics performance and its correlates using data from TIMSS 95 and TIMSS 99. Educational Research and Evaluation, 11: 155-177.
44. O'Loughlin, K. 1994. The assessment of writing by English and ESL teachers. Australian Review of Applied Linguistics, 17: 23-44.
45. Osborne, J. W. (2000). Advantages of hierarchical linear modeling. Practical Assessment, Research & Evaluation, 7 (1). http://PAREonline.net/ getvn.asp?v=7&n=1 (http://PAREonline.net/getvn.asp?v=7&n=1)
46. Raudenbush, S. W., and, Bryk, A. S. 2002. Hierarchical linear models: Applications and data analysis methods, 2nd, Thousand Oaks, CA: Sage.
47. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F., and, Congdon, R. 2004. HLM6: Hierarchical linear and nonlinear modeling, Lincolnwood, IL: Scientific Software International.
48. Raudenbush, S. W., Rowan, B., and, Kang, S.J. 1991. A multilevel, multivariate model for Studying school climate in secondary schools with estimation via the EM algorithm. Journal of Educational Statistics, 16, 4: 295-330.
49. Reise, S. P., and, Duan, N. 2003. " Design issues in multilevel studies ". In Multilevel modeling: Methodological advances, issues, and applications, Edited by: Reise, S. P. and Duan, N. 285-298. Mahwah, NJ: Erlbaum.
50. Rinnert, C., and, Kobayashi, H. 2001. Differing perceptions of EFL writing among readers in Japan. The Modern Language Journal, 85: 189-209.
51. Roux, A. V. D. 2003. " Potentialities and limitations of multilevel analysis in public health and epidemology ". In Methodology and epistemology of multilevel analysis: Approaches from different social sciences, Edited by: Courgeau, D. 93-119. London, UK: Kluwer Academic.
52. Rupp, A. A., Garcia, P., and, Jamieson, J. 2001. Combining multiple regression and CART to understand difficulty in second language reading and listening comprehension test items. International Journal of Testing, 1: 185-216.
53. Sasaki, M., and, Hirose, K. 1999. Development of an analytic rating scale for Japanese L1 writing. Language Testing, 16: 457-478.
54. Shin, J., Espin, C. A., Deno, S. L., and, McConnell, S. 2004. Use of hierarchical linear modeling and curriculum-based measurement for assessing academic growth and instructional factors for students with learning difficulties. Asia Pacific Education Review, 5: 136-148.
55. Shiotsu, T., and, Weir, C. 2007. The relative significance of syntactic knowledge and vocabulary breadth in the prediction of reading comprehension test performance. Language Testing, 24: 99-128.
56. Singer, J. D., and, Willett, J. B. 2003. Applied longitudinal data analysis: Modeling change and event occurrence, Oxford, UK: Oxford University Press.
57. Snellings, P., van Gelderen, A., and, de Glopper, K. 2004. Validating a test of second language written lexical retrieval: A new measure of fluency in written language production. Language Testing, 21: 174-201.
58. Snijders, T. A. B., and, Bosker, R. L. 1999. Multilevel analysis: An introduction to basic and advance multilevel modeling, Thousand Oaks, CA: Sage.
59. Spelberg, H. C. L., de Boer, P., and, van den Bos, K. P. 2000. Item type comparisons of language comprehension tests. Language Testing, 17: 311-322.
60. Stalnaker, J. M. 1936. The problem of the English examination. Educational Record, 17: 45-46.
61. Sweedler-Brown, C. O. 1985. The influence of training and experience on holistic essay evaluation. English Journal, 74: 49-55.
62. Vacc, N. N. 1989. Writing evaluation: Examining four teachers holistic and analytic scores. The Elementary School Journal, 90: 87-95.
63. Weigle, S. C. 2002. Assessing writing, Cambridge, UK: Cambridge University Press.
64. Willms, J. D. 1999. " Basic concepts in hierarchical linear modeling with applications for policy analysis ". In Handbook of education policy, 473-493. New York, NY: Academic Press.
65. Wong, G., and, Mason, W. 1985. The hierarchical logistic regression model for multilevel analysis. Journal of the American Statistical Association, 80: 513-524.
66. Yamada, K. 2009. Lexical patterns in L2 textual gist identification assessment. Language Testing, 26: 101-122.
67. Yang, M., Goldstein, H., Browne, W., and, Woodhouse, G. 2002. Multivariate multilevel analyses of examination results. Journal of the Royal Statistical Society: Series A (Statistics in Society), 165: 137-153.