Comparability of computer-based and paper-and-pencil testing in K-12 reading assessments: A meta-analysis of testing mode effects

Educational and Psychological Measurement

Volumn 68, Issue 1, 2008, Pages 5-24

  Wang, Shudong ^a,b   Jiao, Hong ^a   Young, Michael J ^a   Brooks, Thomas ^a   Olson, John ^a  

^a Harcourt Assessment

^b Harcourt Assessment Inc   (United States)

Author keywords

Comparability of educational test modes; Computer based testing; K 12 reading tests; Meta analysis

Indexed keywords

EID: 37649020581     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164407305592     Document Type: Article

References (61)

1. American Council on Education. (1995). Guidelines for computerized adaptive test development and use in education. Washington, DC: Author.
2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3. American Psychological Association Committee on Professional Standards and Committee on Psychological Tests and Assessment. (1986). Guidelines for computer-based tests and interpretations. Washington, DC: Author.
4. * Arce-Ferrer, A., Lau, C.A., & Griph, G. (2004). Comparison of paper-and-pencil and online versions of Grade 5 mathematics and Grade 8 reading and writing tests. San Antonio, TX: Harcourt Assessment.
5. Association of Test Publishers. (2000). Guidelines for computer-based testing. Washington, DC: Author.
6. Becker, B.J. (1988). Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology, 41, 257-278.
7. Bennett, R.E. (2001). How the Internet will help large-scale assessment reinvent itself. Education Policy Analysis Archive, 9 (5). Retrieved June 20, 2004, from http://epaa.asu.edu/epaa/v9n5.html
8. Bennett, R.E. (2002). Inexorable and inevitable: The continuing story of technology and assessment. Journal of Technology, Learning, and Assessment, 1 (1). Available at http://www.bc.edu/research/intasc/jtla/journal/pdf/v1n1_jtla. pdf
9. Bergstrom, B. (1992, April). Ability measure equivalence of computer adaptive and pencil and paper tests: A research synthesis. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
10. Boo, J., & Vispoel, W.P. (1998, April). Computer versus paper-pencil assessment of educational development: Score comparability and examinee preference. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Quebec, Canada.
11. Bugbee, A.C. (1996). The equivalence of paper-and-pencil and computer-based testing. Journal of Research on Computing in Education, 28, 282-299.
12. Chin, C.H.L., & Donn, J.S. (1991). Effects of computer-based tests on the achievement, anxiety, and attitudes of Grade 10 science students. Educational and Psychological Measurement, 51, 735.
13. Choi, S.W., & Tinkler, T. (2002, April). Evaluating comparability of paper-and-pencil and computer-based assessment in a K-12 setting. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
14. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
15. Dunlop, W.P., Cortina, J.M., Vaslow, J.B., & Burke, M.J. (1996). Meta-analysis of experiments with matched groups or repeated measures design. Psychological Methods, 1, 170-177.
16. * Eignor, D.R. (1993). Deriving comparable scores for computer adaptive and conventional tests: An example using the SAT (Research Report RR-93-55). Princeton, NJ: Educational Testing Service.
17. Evans, L.D., Tannehill, R., & Martin, S. (1995). Children's reading skills: A comparison of traditional and computerized assessment. Behavior Research Methods, Instruments, & Computers, 27 (2). 162-165.
18. Folk, V.G., & Smith, R. (1998, September). Model for delivery of computer-based tests. Paper presented at the ETS-sponsored colloquium on Computer-Based Tests: Building the Foundation for Future Assessments, Philadelphia, PA.
19. Godwin, J. (1999, April). Designing the ACT ESL listening test. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
20. Goldberg, A., Russell, M., & Cook, A. (2003). The effect of computers on student writing: A meta-analysis of studies from 1992 to 2002. Journal of Technology, Learning, and Assessment, 2 (1). Available at http://escholarship. bc.edu/cgi/viewcontent.cgi?article=1007&context=jtla
21. Green, B.F., Bock, R.D., Humphreys, L.G., Linn, R.L., & Reckase, M.D. (1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21, 347-360.
22. Hedges, L.V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128.
23. Hedges, L.V. (1987). How hard is hard science, how soft is soft science: The empirical cumulativeness of research. American Psychologist, 42, 443-455.
24. Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
25. International Test Commission. (2004). International guidelines on computer-based and Internet-delivered testing. Available from http://www.intestcom.org/itc_projects.htm
26. * Ito, K., & Sykes, R.C. (2004, April). Comparability of scores from norm-reference paper-and-pencil and Web-based linear tests for Grades 4-12. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
27. Johnson, M., & Green, S. (2004). On-line assessment: The impact of mode on student performance. Paper presented at the British Educational Research Association Annual Conference, Manchester, UK.
28. Kim, J. (1999, October). Meta-analysis of equivalence of computerized and P&P tests on ability measures. Paper presented at the annual meeting of the Mid-Western Educational Research Association, Chicago, IL.
29. * Kingsbury, G.G. (2002, April). An empirical comparison of achievement level estimates from adaptive tests and paper-and-pencil tests. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
30. * Kingsbury, G.G., & Houser, R.L. (1988, April). A comparison of achievement level estimates from computerized adaptive testing and paper-and-pencil testing. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
31. Klein, S.P., & Hamilton, L. (1999). Large-scale testing: Current practices and new directions (Research Report IP-182). Santa Monica, CA: RAND.
32. Kolen, M.J., & Brennan, R.L. (1995). Test equating: Methods and practices. New York: Springer-Verlag.
33. Lipsey, M.W., & Wilson, D.B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
34. Mazzeo, J., & Harvey, A.L. (1988). The equivalence of scores from automated and conventional educational and psychological tests: A review of the literature (College Board Rep. No. 88-8, ETS RR No. 88-21). Princeton, NJ: Educational Testing Service.
35. McKee, L.M., & Levinson, E.M. (1990). A review of the computerized version of the Self-Directed Search. Career Development Quarterly, 38, 325-333.
36. Mead, A.D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 9, 287-304.
37. Morris, S.B., & DeShon, R.P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105-125.
38. Mueller, D.J., & Wasser, V. (1977). Implications of changing answers on objective test items. Journal of Educational Measurement, 14, 9-14.
39. National Association of State Boards of Education. (2001). Any time, any place, any path, any pace: Taking the lead on e-learning policy. Retrieved June 10, 2003, from http://www.nasbe.org/Organization_Information/e_learning.pdf
40. National Center for Education Statistics. (2000). Teacher use of computers and the Internet in public schools (NCES 2000-090). Retrieved June 10, 2003, from http://nces.ed.gov/pubs2000/2000090.pdf
41. National Commission on Excellence in Education. (1983). A nation at risk: The imperative for educational reform. Retrieved June 10, 2003, from http://www.ed.gov/pubs/NatAtRisk
42. Neuman, G., & Baydoun, R. (1998). Computerization of paper-and-pencil tests: When are they equivalent? Applied Psychological Measurement, 22, 71-83.
43. Park, J. (2003). A test-taker's perspective. Education Week, 22 (35). 15.
44. Parshall, C.G., Spray, J.A., Kalohn, J.C., & Davey, T. (2002). Practical considerations in computer-based testing. New York: Springer.
45. * Poggio, J., Glasnapp, D., Yang, X., Beauchamp, A., & Dunham, M. (2005, April). Assessing reading comprehension based on computerized and paper and pencil formats in a state large scale assessment program. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
46. * Pommerich, M. (2004). Developing computerized versions of paper-and-pencil tests: Mode effects for passage-based tests. Journal of Technology, Learning, and Assessment, 2 (6). Available at http://escholarship. bc.edu/cgi/viewcontent.cgi?article=1002&context=jtla
47. Pommerich, M., & Burden, T. (2000, April). From simulation to application: Examinees react to computerized testing. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA. Available at http://escholarship.bc.edu/cgi/viewcontent.cgi?article= 1002&context=jtla
48. * Pomplun, M., Frey, S., & Becker, D. (2000). The score equivalence of paper-and-pencil and computerized versions of a speeded test of reading comprehension. Educational and Psychological Measurement, 62, 337-354.
49. * Pomplun, M., Frey, S., Becker, D., & Hughes, K. (2002, April). The validity of a computerized measure of reading rate. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
50. Russell, M. (1999). Testing writing on computers: A follow-up study comparing performance on computer and paper. Educational Policy Analysis Archives, 7 (20). Available at http://epaa.asu.edu/epaa/v7n20/
51. Russell, M., & Haney, W. (1997). Testing writing on computers: An experiment comparing student performance on tests conducted via computer and via paper-and-pencil. Educational Policy Analysis Archives, 5 (3). Available from http://olam.ed.asu.edu/epaa/v5n3.html
52. Russell, M., & Plati, T. (2001 a). Effects of computer versus paper administration of a state-mandated writing assessment. Teachers College Record. Available from http://www.tcrecord.org/PrintContent.asp?ContentID=10709
53. Russell, M., & Plati, T. (2001 b). Mode of administration effects on MCAS composition performance for Grade eight and ten. Teachers College Record. Available from http://www.tcrecord.org/Content.asp?ContentID=10709
54. Schmit, M.J., & Ryan, A.M. (1993). Test-taking disposition: A missing link? Journal of Applied Psychology, 77, 624-637.
55. * Schwarz, R.D., Rich, C., & Podrabsky, T. (2003). A DIF analysis of item-level mode effects for computerized and paper-and-pencil tests. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
56. Stenner, A.J. (1996, October). Measuring reading comprehension with the Lexile framework. Paper presented at the California Comparability Symposium, Burlingame, CA.
57. Vispoel, W.P., Wang, T., de la Torre, R., Bleiler, T., & Dings, J. (1992, April). How review options and administration mode influence scores on computerized vocabulary tests. Paper presented at the annual meeting of the AERA, San Francisco, CA. (ERIC Document Reproduction Service No. ED 346 161)
58. Wang, S., Jiao, H., Young, M.J., Brooks, T.E., & Olson, J. (2007). A meta-analysis of testing mode effects in Grade K-12 mathematics tests. Educational and Psychological Measurement, 67, 219-238.
59. Wang, S., Newman, L., & Witt, E.A. (2000). AT&T aptitude test equivalence study: A comparison of computer and paper-and-pencil employment examinations. Bala Cynwyd, PA: Harcourt Assessment System.
60. * Wang, S., Young, M.J., & Brooks, T.E. (2004). Administration mode comparability study for Stanford Diagnostic Reading and Mathematics Tests. San Antonio, TX: Harcourt Assessment.
61. Wise, S.L., & Plake, B.S. (1989). Research on the effects of administering tests via computers. Educational Measurement: Issues and Practice, 8 (3). 5-10.