Equivalence of reading and listening comprehension across test media

Educational and Psychological Measurement

Volumn 71, Issue 5, 2011, Pages 849-869

  Schroeders, Ulrich ^a   Wilhelm, Oliver ^a  

^a HUMBOLDT UNIVERSITY   (Germany)

Author keywords

CBA; comparability; equivalence; MGCFA

Indexed keywords

EID: 80053189000     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164410391468     Document Type: Review

References (45)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2. Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least square estimation in confirmatory factor analysis. Structural Equation Modeling, 13, 186-203.
3. Blackhurst, A. (2005). Listening, reading and writing on computer-based and paper-based versions of IELTS. Cambridge ESOL Research Notes, 21, 14-17. Retrieved from http://www. cambridgeesol.org/rs_notes/rs_nts21.pdf
4. Bollen, K. A. (1989). Structural equations with latent variables. Oxford, England: John Wiley.
5. Bridgeman, B., Lennon, M. L., & Jackenthal, A. (2003). Effects of screen size, screen resolution and display rate on computer-based test performance. Applied Measurement in Education, 16, 191-205.
6. Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (Eds.). (2008). Building a validity argument for the Test of English as a Foreign Language. New York, NY: Routledge.
7. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.
8. Choi, I.-C., Kim, K. S., & Boo, J. (2003). Comparability of a paper-based language test and a computer-based language test. Language Testing, 20, 295-320.
9. Clausing, C. S., & Schmitt, D. R. (1990). Paper versus CRT: Are reading rate and comprehension affected? In Proceedings of selected paper presentations at the convention of the Association for Educational Communications and Technology. (ERIC Document Reproduction Service No. ED323924)
10. Coniam, D. (2006). Evaluating computer-based and paper-based versions of an English language listening test. ReCALL Journal, 18, 193-211.
11. Deshon, R. P. (2004). Measures are not invariant across groups with error variance homogeneity. Psychology Science, 46, 137-149.
12. Eid, M., Lischetzke, T., Nussbeck, F. W., & Trierweiler, L. I. (2003). Separating trait effects from trait-specific method effects in multitrait-multimethod models: A multiple-indicator CT-C(M-1) model. Psychological Methods, 8, 38-60.
13. Educational Testing Service. (2001). Computer-based TOEFL. Score user guide. Retrieved from http://www.ets.org/Media/Tests/TOEFL/pdf/989551.pdf
14. Farcot, M., & Latour, T. (2009). Transitioning to computer-based assessments: A question of costs. In F. Scheuermann & J. Björnsson (Eds.), The transition to computer-based assessment (pp. 108-116). JRC Scientific and Technical Reports. Retrieved from http://crell.jrc.it/RP/reporttransition.pdf
15. Halldórsson, A. M., McKelvie, P., & Björnsson, J. K. (2009). Are Icelandic boys really better on computerized tests than conventional ones? Interaction between gender, test modality and test performance. In F. Scheuermann & J. K. Björnsson (Eds.), The transition to computer-based assessment (pp. 178-193). JRC Scientific and Technical Reports. Retrieved from http://crell.jrc.it/RP/reporttransition.pdf
16. Higgins, J., Russell, M., & Hoffmann, T. (2005). Examining the effect of computer-based passage presentation on reading test performance. Journal of Technology, Learning, and Assessment, 3(4). Retrieved from http://escholarship.bc.edu/jtla/
17. Horkay, N., Bennett, R. E., Allen, N., & Kaplan, B. (2005). Online assessment in writing. In B. Sandene, N. Horkay, R. E. Bennett, N. Allen, J. Braswell, B. Kaplan, & A. Oranje (Eds.), Online assessment in mathematics and writing: Reports from the NAEP Technology-Based Assessment Project (NCES 2005-457). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
18. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
19. Kim, D.-H., & Huynh, H. (2008). Computer-based and paper-and-pencil administration mode effects on a statewide end-of-course English test. Educational and Psychological Measurement, 68, 554-570.
20. Leeson, H. V. (2006). The mode effect: A literature review of human and technological issues in computerized testing. International Journal of Testing, 6, 1-24.
21. Lubke, G. H., Dolan, C. V., Kelderman, H., & Mellenbergh, G. J. (2003). On the relationship between sources of within and between-group differences and measurement invariance in the common factor model. Intelligence, 31, 543-566.
22. McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.
23. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525-543.
24. Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39, 479-515.
25. Muthén, B. O., & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus (Mplus Web Notes: No. 4). Retrieved from http://www.statmodel.com/examples/webnote.shtm
26. Muthén, B. O., & Christoffersson, A. (1981). Simultaneous factor analysis of dichotomous variables in several groups. Psychometrika, 46, 407-419.
27. Muthén, L. K., & Muthén, B. O. (2007). Mplus user's guide. Los Angeles, CA: Muthén & Muthén.
28. Muthén, L. K., & Muthén, B. O. (2009). Mplus (Version 5.21) [Computer software]. Los Angeles, CA: Muthén & Muthén.
29. Neuman, G., & Baydoun, R. (1998). Computerization of paper-and-pencil tests: When are they equivalent? Applied Psychological Measurement, 22, 71-83.
30. Overton, R. C., Taylor, L. R., Zickar, M. J., & Harms, H. J. (1996). The pen-based computer as an alternative platform for test administration. Personnel Psychology, 49, 455-464.
31. Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: Mode effects for passage-based tests. Journal of Technology, Learning, and Assessment, 2(6). Retrieved from http://escholarship.bc.edu/jtla/
32. Pomplun, M., Frey, S., & Becker, D. F. (2002). The score equivalence of paper-and-pencil and computerized versions of a speeded test of reading comprehension. Educational and Psychological Measurement, 62, 337-354.
33. Raju, N. S., Laffitte, L. J., & Byrne, B. M. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87, 517-529.
34. Revuelta, J., Ximénez, M. C., & Olea, J. (2003). Psychometric and psychological effects of item selection and review on computerized testing. Educational and Psychological Measurement, 63, 791-808.
35. Rupp, A. A., Vock, M., Harsch, C., & Köller, O. (2008). Developing standards-based assessment tasks for English as a first foreign language. Context, processes, and outcomes in Germany. Münster, Germany: Waxmann.
36. Sandene, B., Horkay, N., Bennett, R. E., Allen, N. Braswell, J., Kaplan, B., & Oranje, A. (Eds.). (2005). Online assessment in mathematics and writing: Reports from the NAEP technologybased assessment project (NCES 2005-457). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
37. Schroeders, U. (2009). Testing for equivalence of test data across media. In: F. Scheuermann & J. Björnsson (Eds.), The transition to computer-based assessment (pp. 164-170). JRC Scientific and Technical Reports. Retrieved from http://crell.jrc.it/RP/reporttransition.pdf
38. Schroeders, U., & Wilhelm, O. (2010). Testing reasoning ability with handheld computers, notebooks, and paper and pencil. European Journal of Psychological Assessment, 26, 284-292.
39. van Lent, G. (2008). Important considerations in e-assessment. In F. Scheuermann & A. Guimarães Pereira (Eds.), Towards a research agenda on computer-based assessment (pp. 97-103). Ispra, Italy: European Commission, Joint Research Centre.
40. Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the MI literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69.
41. Venter, A., & Maxwell, S. E. (1999). Maximizing power in randomized designs when sample size is small. In R. Hoyle (Ed.), Statistical strategies for small sample research (pp. 31-58). Thousand Oaks, CA: Sage.
42. Wang, S., Jiao, H., Young, M. J., Brooks, T., & Olson, J. (2008). Comparability of computerbased and paper-and-pencil testing in K-12 reading assessments: A meta-analysis of testing mode effects. Educational and Psychological Measurement, 68, 5-24.
43. Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research, and Evaluation, 12(3). Retrieved from http:// pareonline.net/pdf/v12n3.pdf
44. Yu, C. Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes (Unpublished doctoral dissertation). University of California, Los Angeles.
45. Zinbarg, R., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach's α, Revelle's β, and McDonald's ω: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123-133.