Automatic scoring of non-native spontaneous speech in tests of spoken English

Speech Communication

Volumn 51, Issue 10, 2009, Pages 883-895

  Zechner, Klaus ^a   Higgins, Derrick ^a   Xi, Xiaoming ^a   Williamson, David M ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Automatic scoring; Scoring of spontaneous speech; Speaking assessment; Speech scoring; Spoken language scoring

Indexed keywords

AUTOMATIC SCORING; AUTOMATIC SPEECH RECOGNITION; CLASSIFICATION AND REGRESSION TREE; COMPLETE SETS; CORRELATION COEFFICIENT; EMPIRICAL CORRELATIONS; ENGLISH-AS-A-FOREIGN-LANGUAGE; FEATURE COMPUTATION; FIELD STUDIES; INTERNET BASED; MULTIPLE REGRESSIONS; NON-NATIVE; OPERATIONAL USE; RESEARCH AND DEVELOPMENT PROGRAMS; SCORING MODELS; SCORING OF SPONTANEOUS SPEECH; SPEAKING ASSESSMENT; SPEECH DATA; SPEECH RECOGNIZER; SPOKEN LANGUAGE SCORING; SPONTANEOUS SPEECH; TEST DEVELOPMENT;

COMMUNICATION; FEATURE EXTRACTION; LINGUISTICS; REGRESSION ANALYSIS; TESTING;

SPEECH RECOGNITION;

EID: 67650710841     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2009.04.009     Document Type: Article

References (29)

1. Attali, Y., Burstein, J., 2004. Automated essay scoring with e-rater V.2.0. Presented at the Annual Meeting of the International Association for Educational Assessment, Philadelphia, PA.
2. Attali, Y., Burstein, J., 2005. Automated essay scoring with e-rater V.2.0 (ETS RR-04-45). Educational Testing Service, Princeton, NJ.
3. Bachman L.F. Fundamental Considerations in Language Testing (1990), Oxford University Press, Oxford
4. Bachman L.F., and Palmer A. Language Testing in Practice: Designing and Developing Useful Language Tests (1996), Oxford University, Oxford
5. Bernstein J. PhonePass Testing: Structure and Construct (1999), Ordinate Corporation, Menlo Park, CA
6. Bernstein, J., DeJong, J., Pisoni, D., Townshend, B., 2000. Two experiments in automatic scoring of spoken language proficiency. In: Proceedings of InSTILL2000, Dundee, Scotland.
7. Brieman L., Jerome F., Olshen R., and Stone C. Classification and Regression Trees (1984), Wadsworth, Pacific Grove
8. Burstein, J., Kukich, K., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., Lu, C., Nolan, J., Rock, D., Wolff, S., 1998. Computer analysis of essay content for automated score prediction: a prototype automated scoring system for GMAT analytical writing assessment (ETS RR-98-15). Educational Testing Service, Princeton, NJ.
9. Chodorow, M., Burstein, J., 2004. Beyond essay length: evaluating e-rater's performance on TOEFL essays (TOEFL Research Report No. RR-73, ETS RR-04-04). Educational Testing Service, Princeton, NJ.
10. Clauser B.E., Margolis M.J., Clyman S.G., and Ross L.P. Development of automated scoring algorithms for complex performance assessments: a comparison of two approaches. Journal of Educational Measurement 34 (1997) 141-161
11. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70 (1968) 213-220
12. Cucchiarini, C., Strik, H., Boves, L., 1997a. Automatic evaluation of Dutch pronunciation by using speech recognition technology. Paper Presented at the IEEE Automatic Speech Recognition and Understanding Workshop, Santa Barbara, CA.
13. Cucchiarini, C., Strik, H., Boves, L., 1997b. Using speech recognition technology to assess foreign speakers' pronunciation of Dutch. Paper Presented at the Third International Symposium on the Acquisition of Second Language Speech: NEW SOUNDS 97, Klagenfurt, Austria.
14. Cucchiarini C., Strik H., and Boves L. Different aspects of expert pronunciation quality ratings and their relation to scores produced by speech recognition algorithms. Speech Communication 30 2-3 (2000) 109-119
15. Cucchiarini C., Strik H., and Boves L. Quantitative assessment of second language learners' fluency by means of automatic speech recognition technology. Journal of the Acoustical Society of America 107 (2000) 989-999
16. Cucchiarini C., Strik H., and Boves L. Quantitative assessment of second language learners' fluency: comparisons between read and spontaneous speech. Journal of the Acoustical Society of America 111 6 (2002) 2862-2873
17. Franco, H., Abrash, V., Precoda, K., Bratt, H., Rao, R., Butzberger, J., 2000a. The SRI EduSpeak system: recognition and pronunciation scoring for language learning. In: Proceedings of InSTiLL-2000 (Intelligent Speech Technology in Language Learning), Dundee, Scotland.
18. Franco H., Neumeyer L., Digalakis V., and Ronen O. Combination of machine scores for automatic grading of pronunciation quality. Speech Communication 30 (2000) 121-130
19. Landauer T.K., and Dumais S.T. A solution to Plato's problem: the Latent Semantic Analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review 104 (1997) 211-240
20. Leacock C. Scoring free-responses automatically: a case study of a large-scale assessment. Examens 1 3 (2004)
21. Leacock C., and Chodorow M. C-rater: scoring of short-answer questions. Computers and the Humanities 37 4 (2003) 389-405
22. Linguistic Data Consortium (LDC), 1997. HUB-4 Broadcast News corpus (English).
23. North B. The Development of a Common Framework Scale of Language Proficiency (2000), Peter Lang, New York, NY
24. Page E.B. The imminence of grading essays by computer. Phi Delta Kappan 47 (1966) 238-243
25. Rudner, L.M., Garcia, V., Welch, C., 2006. An evaluation of the Intellimetric essay scoring system. Journal of Technology, Learning and Assessment 4 (4). Retrieved from .
26. Steinberg D., and Colla P. CART: Tree-Structured Non-Parametric Data Analysis (1995), Salford Systems, San Diego, CA
27. Williamson D.M., Bejar I.I., and Hone A.S. "Mental model" comparison of automated and human scoring. Journal of Educational Measurement 36 (1999) 158-184
28. ® Academic Speaking Test (TAST). TOEFL iBT Research Report No. TOEFLiBT-01.
29. Zechner, K., Bejar, I., 2006. Towards automatic scoring of non-native spontaneous speech. In: Proceedings of the 2006 Conference on Human Language Technology and the North-American Association for Computational Linguistics (HLT-NAACL-06), New York, NY.