AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATERSM V1.0

ETS Research Report Series

Volumn 2008, Issue 2, 2008, Pages i-102

  Xi, Xiaoming ^a   Higgins, Derrick ^a   Zechner, Klaus ^a   Williamson, David M ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

automated scoring; automatic speech processing; automatic speech recognition; TOEFL; TOEFL Practice Online; validity argument

Indexed keywords

EID: 78049530860     PISSN: None     EISSN: 23308516     Source Type: Journal    
DOI: 10.1002/j.2333-8504.2008.tb02148.x     Document Type: Article

References (77)

1. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2.0. Journal of Technology, Learning, and Assessment, 4(3), 1–34.
2. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford, England: Oxford University Press.
3. Bachman, L. F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1–34.
4. Bachman, L. F., & Palmer, A. (1996). Language testing in practice: Designing and developing useful language tests. Oxford, England: Oxford University Press.
5. Bejar, I. I. (1991). A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology, 76(4), 522–532.
6. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice, 17(4), 9–17.
7. Bennett, R. E., Sebrechts, E., & Marc, M. (1994). The accuracy of automatic qualitative analyses of constructed-response solutions to algebra word problems (ETS Research Rep. No. RR-94-04). Princeton, NJ: ETS.
8. Bernstein, J. (1999). PhonePass testing: Structure and construct. Menlo Park, CA: Ordinate.
9. Braun, H. I., Bennett, R. E., Frye, D., & Soloway, E. (1990). Scoring constructed responses using expert systems. Journal of Educational Measurement, 27(2), 93–108.
10. Brieman, L., Jerome F., Olshen, R., & Stone, C. (1984). Classification and regression trees. Pacific Grove, CA: Wadsworth.
11. Brown, A., Iwashita, N., & McNamara, T. (2005). An examination of rater orientations and test-taker performance on English-for-academic-purposes speaking tasks (TOEFL Monograph Series No. 29 MS-29). Princeton, NJ: ETS.
12. Burstein, J., Kukich, K., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., et al. (1998). Computer analysis of essay content for automated score prediction: A prototype automated scoring system for GMAT analytical writing assessment (ETS Research Rep. No. RR-98-15). Princeton, NJ: ETS.
13. Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 8–24.
14. Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (Eds.). (2008). Building a validity argument for the Test of English as a Foreign Language. Mahwah, NJ: Lawrence Erlbaum.
15. Chodorow, M., & Burstein, J. (2004). Beyond essay length: Evaluating e-rater's performance on TOEFL essays (TOEFL Research Rep. No. RR-73). Princeton, NJ: ETS.
16. Clauser, B. E., Subhiyah R., Nungester R. J., Ripkey D. R., Clyman, S. G., & McKinley D. (1995). Scoring a performance-based assessment by modeling the judgments of experts. Journal of Educational Measurement, 32(4), 397–415.
17. Clauser, B. E., Kane, M. T., & Swanson, D. B. (2002). Validity issues for performance-based tests scored with computer-automated scoring systems. Applied Measurement in Education, 15(4), 413–432.
18. Clauser, B. E., Margolis, M. J., Clyman, S. G., & Ross, L. P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal of Educational Measurement, 34(2), 141–161.
19. Cohen J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–20.
20. Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443–507). Washington, DC: American Council on Education.
21. Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer & H. Braun (Eds.), Test validity (pp. 3–17). Hillsdale, NJ: Lawrence Erlbaum.
22. Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
23. Cucchiarini, C., Strik, H., & Boves, L. (2000). Quantitative assessment of second language learners' fluency by means of automatic speech recognition technology. Acoustical Society of America, 107(2), 989–999.
24. Cucchiarini, C., Strik, H., & Boves, L. (2002). Quantitative assessment of second language learners' fluency: Comparisons between read and spontaneous speech. Journal of the Acoustical Society of America, 111(6), 2862–2873.
25. Cureton, E. F. (1951). Validity. In E. F. Lindquist (Ed.), Educational measurement (pp. 621–694). Washington, DC: American Council on Education.
26. Dechert, H. W. (1980). Pauses and intonation as indicators of verbal planning in second-language speech productions: Two examples from a case study. In H. W. Dechert & M. Raupach (Eds.), Temporal variables in speech (pp. 271–285). The Hague, The Netherlands: Mouton.
27. Derwing, T., & Munro, M. J. (1997). Accent, comprehensibility, and intelligibility: Evidence from four L1s. Studies in Second Language Acquisition, 19(1), 1–16.
28. Deschamps, A. (1980). The syntactical distribution of pauses in English spoken as a second language by French students. In H. W. Dechert & M. Raupach (Eds.), Temporal variables in speech (pp. 255–262). The Hague, The Netherlands: Mouton.
29. ETS. (n.d.). TOEFL iBT scores set by universities and other score users. Retrieved September 25, 2008, from http://www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=031e4e63dcc85010VgnVCM10000022f95190RCRD&vgnextchannel=333bd898c84f4010VgnVCM10000022f95190RCRD.
30. Franco, H., Abrash, V., Precoda, K., Bratt, H., Rao, R., & Butzberger, J. (2000, August). The SRI EduSpeak system: Recognition and pronunciation scoring for language learning. Paper presented at the annual conference of Intelligent Speech Technology in Language Learning, Dundee, Scotland.
31. FFmpeg. (n.d.). Home page. Available from http://ffmpeg.mplayerhq.hu/.
32. Fiscus, J., Garofolo, J., Praybocki, M., Fisher, W., & Pallett, D. (1997). 1997 English broadcast news speech (HUB-4; English). Philadelphia: Linguistic Data Consortium.
33. Frank, E., Wang, Y. S., Inglis, G. H., Holmes, G., & Witten, I. H. (1998). Using model trees for classification. Machine Learning, 32(1), 63–76.
34. Freed, B. (Ed.). (1995). Second language acquisition in a study abroad context. Philadelphia: John Benjamins.
35. Haberman, S. J. (1979). Analysis of qualitative data: Vol. 2, new developments. New York: Academic Press.
36. Hansen, L., Gardner, J., & Pollard, J. (1998). The measurement of fluency in a second language: Evidence from the acquisition and attrition of Japanese. In B. Visgatis (Ed.), On JALT '97: Trends and transitions. Proceedings of the JALT 1997 Conference on Language Teaching and Learning (pp. 37–46). Tokyo: The Japan Association for Language Teaching.
37. Jurafsky, D., & Martin, J. H. (2000). Speech and language processing. Upper Saddle River, NJ: Prentice-Hall.
38. Kaplan, R. M., & Bennett, R. E. (1994). Using the free-response scoring tool to automatically score the formulating-hypothesis item (ETS GRE Technical Rep. GRE 90-02b). Princeton, NJ: ETS.
39. Kane, M. T. (1992). An argument-based approach to validity, Psychological Bulletin, 112(3), 527–535.
40. Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319–342.
41. Kane, M. T. (2002). Validating high-stakes testing programs. Educational Measurement: Issues and Practice, 21(1), 31–35.
42. Kane, M. T. (2004). Certification testing as an illustration of argument-based validation. Measurement: Interdisciplinary Research and Perspectives, 2(3), 135–170.
43. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed). Educational measurement (4th ed., pp. 17–64). Washington, DC: American Council on Education/Praeger.
44. Kane, M. T., Crooks, T., & Cohen, A. (1999). Validating measures of performance. Educational Measurement: Issues and Practice, 18(2), 5–17.
45. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240.
46. Leacock, C., & Chodorow, M. (2003). C-rater: Scoring of short-answer questions. Computers and the Humanities, 37(4), 389–405.
47. Lennon, P. (1984). Retelling a story in English as a second language. In H. W. Dechert, D. Mohle, & M. Raupach (Eds.), Second language productions (pp. 50–68). Tübingen, Germany: Gunter Narr.
48. Lennon, P. (1990). Investigating fluency in EFL: A quantitative approach. Language Learning, 40(3), 387–417.
49. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8), 707–710.
50. Malvern, D. D., & Richards, B. J. (2000). Investigating accommodation in language proficiency interviews using a new measure of lexical diversity. Language Testing, 19(1), 85–104.
51. Manning, C., & Schuetze, H. (1999). Foundations of statistical natural language processing. Cambridge, MA: MIT Press.
52. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education/Macmillan.
53. Mislevy, R. J., Steinberg, L. S., Almond, R. G., & Lukas, J. F. (2006). Concepts, terminology, and basic models of evidence-centered design. In D. M. Williamson, R. J. Mislevy, & I. I. Bejar (Eds.), Automated scoring for complex constructed response tasks in computer based testing (pp. 15–48). Mahwah, NJ: Lawrence Erlbaum.
54. Moehle, D. (1984). A comparison of the second language speech of different native speakers. In H. W. Dechert, D. Mohle, & M. Raupach, (Eds.), Second language productions (pp. 26–49). Tübingen, Germany: Gunter Narr.
55. Munro, M., & Derwing, T. (2001). Modeling perceptions of the accentedness and comprehensibility of L2 speech: The role of speaking rate. Studies in Second Language Acquisition, 23(4), 451–468
56. Page, E. B. (1966). The imminence of grading essays by computer. Phi Delta Kappan, 47(1), 238–243.
57. Page, E. B., & Petersen, N. (1995). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76(7), 561–565.
58. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference (2nd ed.). San Francisco: Morgan Kaufmann.
59. Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 37(2), 257–286.
60. Rabiner, L., & Juang, B.-H. (1993). Fundamentals of speech recognition. Englewood Cliffs, NJ: Prentice Hall.
61. Raupach, M. (1980). Temporal variables in first and second language speech production. In H. W. Dechert & M. Raupach (Eds.), Temporal variables in speech (pp. 271–285). The Hague, The Netherlands: Mouton.
62. Read, J. & Nation, P. (2004). An investigation of the lexical dimension of the IELTS speaking test. Canberra, Australia: International English Language Testing System.
63. Riggenbach, H. (1991). Toward an understanding of fluency: A microanalysis of nonnative speaker conversations source. Discourse Processes, 14(4), 423–441.
64. Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of the Intellimetric essay scoring system. Journal of Technology, Learning and Assessment, 4(4). Available from http://escholarship.bc.edu/jtla/.
65. Sebrechts, M. M., Bennett, R. E., & Donald, R. A. (1991). Machine-scorable complex constructed-response quantitative items: Agreement between expert system and human raters' scores (ETS Research Rep. No. RR-91-11). Princeton, NJ: ETS.
66. Steinberg, D., & Colla, P. (1995). CART: Tree-structured non-parametric data analysis. San Diego, CA: Salford Systems.
67. Steinberg, D., & Colla, P. (1997). CART – Classification and regression trees. San Diego, CA: Salford Systems.
68. Strik, H., & Cucchiarini, C. (1999). Automatic assessment of second language learners' fluency. In Proceedings of the 14th international congress of phonetic sciences (pp. 759–762). Berkeley, CA: University of California, Berkeley.
69. Toulmin, S. E. (2003). The uses of argument (updated ed.). Cambridge, England: Cambridge University Press.
70. Towell, R. (1987). Variability and progress in the language development of advanced learners of a foreign language. In R. Ellis (Ed.), Second language acquisition in context (pp. 113–127). Toronto, Ontario, Canada: Prentice Hall.
71. Wainer, H. (1976). Estimating coefficients in linear models: It don't make no nevermind. Psychological Bulletin, 83(2), 213–217.
72. Williamson, D. M., Bejar, I. I., & Hone, A. S. (1999). “Mental model” comparison of automated and human scoring. Journal of Educational Measurement, 36(2), 158–184.
73. Wood, D. (2001). In search of fluency: What is it and how can we teach it? Canadian Modern Language Review, 57(4), 573–589.
74. Xi, X., & Mollaun, P. (2006). Investigating the utility of analytic scoring for the TOEFL Academic Speaking Test (TAST) (TOEFL iBT Research Series. No. 1). Princeton, NJ: ETS.
75. Xi, X., Zechner, K., & Bejar, I. I. (2006, April). Extracting meaningful speech features to support diagnostic feedback: A construct-driven approach to automated scoring. Symposium paper presented at the annual conference of National Council on Measurement in Education, San Francisco.
76. Yang, Y., Buckendahl, C. W., Juszkiewicz, P. J., & Bhola, D. S. (2002). A review of strategies for validating computer-automated scoring. Applied Measurement in Education, 15(4), 391–412.
77. Zechner, K., Bejar, I. I., & Hemat, R. (2007). Towards an understanding of the role of speech recognition in non-native speech assessment (ETS Research Rep. No. RR-07-02). Princeton, NJ: ETS.