Developments and challenges in the use of computer-based testing for assessing second language ability

Modern Language Journal

Volumn 93, Issue SUPPL. 1, 2009, Pages 836-847

  Ockey, Gary J ^a  

^a KANDA UNIVERSITY OF INTERNATIONAL STUDIES   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 71849102954     PISSN: 00267902     EISSN: 15404781     Source Type: Journal    
DOI: 10.1111/j.1540-4781.2009.00976.x     Document Type: Article

References (57)

1. Alderson J C. Assessing reading 2000, Cambridge, Cambridge University Press
2. Alderson J C, Huhta A. The development of a suite of computer-based diagnostic tests based on the Common European framework. Language Testing 2005, 22:301-320.
3. Attali Y, Burstein J. Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment 2005, 4(3). http://www.jtla.org, Retrieved October 10, 2008, from
4. Bachman L F, Palmer A S. Language testing in practice 1996, Oxford, Oxford University Press
5. Bonk W J, Ockey G J. A many-facet Rasch analysis of the second language group oral discussion task. Language Testing 2003, 20:89-110.
6. Buck G. Assessing listening 2001, Cambridge, Cambridge University Press
7. Burstein J, Kukich K, Wolff S, Lu C, Chodorow M. Computer analysis of essays 1998, April, Paper presented at Proceedings of the NCME symposium on automated scoring, Montreal, Canada
8. Carr N T, Ducate L, Arnold N. Computer-based testing: Prospects for innovative assessment. Calling on CALL: From theory and research to new directions in foreign language teaching 2006, 289-312. In, Eds, CALICO Monograph Series 5, pp, San Marcos, TX, CALICO
9. Chapelle C, Douglas D. Assessing language through computer technology 2006, Cambridge, University of Cambridge Press
10. Choi I -C, Kim K, Boo J. Comparability of a paper-based language test and a computer-based language test. Language Testing 2003, 20:295-320.
11. Chun C W. Commentary: An analysis of a language test for employment: The authenticity of the PhonePass test. Language Assessment Quarterly 2006, 3:295-306.
12. Condon W, Ericsson P, Haswell R. Why less is not more: What we lose by letting a computer score writing samples. Machine scoring of student essays: Truth and consequences 2006, 209-220. In, Eds, pp, Logan, Utah State University Press
13. Coniam D. Evaluating computer-based and paper-based versions of an English-language listening test. ReCALL 2006, 18:193-211.
14. Davies A, Brown A, Elder C, Hill K, Lumley T, McNamara T. Studies in language testing: Dictionary of language testing 1999, Cambridge, Cambridge University Press
15. DIALANG http://www.dialang.org, DIALANG, n.d., Retrieved September 22, 2008, from
16. Dikli S. An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment 2006, (1). http://escholarship.bc.edu/jtla/vol5/1, Retrieved November 2, 2008, from
17. Douglas D, Hegelheimer V. Assessing language using computer technology. Annual Review of Applied Linguistics 2007, 27:115-132.
18. Downey R, Farhady H, Present-Thomas H, Suzuki M, Van Moere A. Evaluation of the usefulness of the Versant for English Test: A response. Language Assessment Quarterly 2008, 5:160-167.
19. GRE details: Test takers 2008a, http://www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=ebd42d3631df4010VgnVCM10000022f95190RCRD&vgnextchannel=54c846f1674f4010VgnVCM10000022f95190RCRD#How_the_Sections_of_the_General_Test_are_Scored, Educational Testing Service, Retrieved November 20, 2008, from
20. TOEFL iBT tips: How to prepare for the next generation TOEFL test and communicate with confidence 2008b, http://ftp.ets.org/pub/toefl/TOEFL_Tips.pdf, Educational Testing Service, Retrieved June 10, 2008, from
21. Embretson S, Reise S. Item response theory for psychologists 2000, Mahwah, NJ, Erlbaum
22. Enright M, Bridgeman B, Cline F. Prototyping a test design for a New TOEFL 2002, April, Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA
23. Ericsson P, Ericsson P, Haswell R. The meaning of meaning: Is a paragraph more than an equation. Machine scoring of student essays: Truth and consequences 2006, 28-37. In, Eds, pp, Logan, Utah State University Press
24. Fulcher G. Testing second language speaking. 2003, London, Pearson Longman
25. Garrett N. Technology in the service of language learning: Trends and issues. Modern Language Journal 1991, 75:74-101.
26. Ginther A. Context and content visuals and performance on listening comprehension stimuli. Language Testing 2002, 19:133-167.
27. Green B, Wainer H, Messick S. The promise of tailored tests. Principals of modern psychological measurement 1983, 69-80. In, Eds, pp, Hillsdale, NJ, Erlbaum
28. Gruba P. A comparison study of audio and video presentation modes in tests of ESL listening comprehension. 1989, Unpublished master's thesis, University of California, Los Angeles
29. Gruba P. The role of digital video media in second language listening comprehension 1999, Unpublished doctoral dissertation, University of Melbourne, Australia
30. Hughes A. Testing for language teachers 1989, Cambridge, Cambridge University Press
31. Jamieson J. Trends in computer-based second language assessment. Annual Review of Applied Linguistics 2005, 25:228-242.
32. Jurafsky D, Martin J. Speech and language processing: An introduction to natural language processing, speech recognition, and computational linguistics 2008, 2nd ed, New York, Prentice Hall
33. Keller L, Swaminathan H, Sireci S. Evaluating scoring procedures for context-dependent item sets. Applied Measurement in Education 2003, 16:207-222.
34. Kondo-Brown K. A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing 2002, 19:3-31.
35. Lado R. Language testing: The construction and use of foreign language tests 1961, London, Longman
36. Lazaraton A. Interlocutor support in oral proficiency interviews: The case of CASE. Language Testing 1996, 13:151-172.
37. Lee Y -W, Gentile C, Kantor R. Analytic scoring of TOEFL CBT essays: Scores from humans and e-rater 2008, http://www.ets.org, Educational Testing Service. TOEFL Research Report RR-81. Retrieved May 12, 2009, from
38. Lord F M. Robbins-Munro procedures for tailored testing. Educational and Psychological Measurement 1971, 31:3-31.
39. Lord F M. Applications of item response theory to practical testing problems 1980, Hillsdale, NJ, Erlbaum
40. Luoma S. Assessing speaking 2004, Cambridge, Cambridge University Press
41. Ockey G J. Construct implications of including still image or video in computer-based listening tests. Language Testing 2007, 24:517-537.
42. Ockey G J. The effects of group members' personalities on a test taker's L2 group oral discussion test scores. Language Testing 2009, 26:161-186.
43. SET-10 test description validation summary 2004, http://www.ordinate.com, Ordinate, Retrieved February 25, 2005, from
44. Osterlind J. Modern measurement: Theory, principles, and applications of mental appraisal 2006, Upper Saddle River, NJ, Pearson Merrill Prentice Hall
45. Versant English Test: Test description and validation report 2008, http://www.versanttest.co.uk/pdf/ValidationReport.pdf, Pearson Education, Retrieved October 25, 2008, from
46. Rover C. Web-based language testing. Language Learning & Technology 2001, 5(2):84-94.
47. Sawaki Y. Comparability of conventional and computerized tests of reading in a second language. Language Learning & Technology 2001, 5(2):38-59.
48. Sawaki Y, Stricker L, Oranje A. Factor structure of the TOEFL Internet-based Test (iBT): Exploration in a field trial sample 2008, http://www.ets.org/Media/Research/pdf/RR-08-09.pdf, Educational Testing Service. TOEFL Research Report 08-09. Retrieved November 2, 2008, from
49. Shi L. Native and nonnative-speaking EFL teachers' evaluation of Chinese students' English writing. Language Testing 2001, 18:303-325.
50. Shohamy E, Gordon C, Kraemer R. The effect of raters' background and training on the reliability of direct writing tests. Modern Language Journal 1992, 76:27-33.
51. Taylor C, Kirsch I, Eignor D, Jamieson J. Examining the relationship between computer familiarity and performance on computer-based language tasks. Language Learning 1999, 49:219-274.
52. Wagner E. Are they watching? Test-taker viewing behavior during an L2 video listening test. Language Learning & Technology 2007, 9(2):16-23.
53. Wainer H. Computerized adaptive testing: A primer 1990, Hillsdale, NJ, Erlbaum
54. Winke P, Howell S, Hricko M. Online assessment of foreign language proficiency: Meeting development, design, and delivery challenges. Online assessment and measurement: Case studies from teacher education, K-12: and corporate 2006, 82-97. In, Eds, pp, London, Information Science Publishing
55. A D-minus for computer exams 2002, http://wired.com/culture/education/news/2002/08/54459, Wired, Retrieved May 15, 2009, from
56. Xi X, Chapelle C A, Chung Y R, Xu J. What and how much evidence do we need? Critical considerations in validating an automated scoring system. Towards adaptive CALL: Natural language processing for diagnostic language assessment 2008, 102-114. In, Eds, pp, Ames, Iowa State University
57. Zechner K, Bejar I, Hemat R. Toward an understanding of the role of speech recognition in nonnative speech assessment 2007, http://www.ets.org/Media/Research/pdf/RR-07-02.pdf, iBT Research report RR-07-02, Princeton, NJ, Educational Testing Service, Retrieved November 1, 2008, from