Applications of text analysis tools for spoken response grading

Language Learning and Technology

Volumn 17, Issue 2, 2013, Pages 171-192

  Crossley, Scott ^a   McNamara, Danielle ^b  

^a GEORGIA STATE UNIVERSITY   (United States)

^b ARIZONA STATE UNIVERSITY   (United States)

Author keywords

Computational linguistics; Corpus linguistics; Language testing; Machine learning; Speaking proficiency

Indexed keywords

EID: 84878513315     PISSN: None     EISSN: 10943501     Source Type: Journal    
DOI: None     Document Type: Article

References (55)

1. Adams, M. L. (1980). Five co-occurring factors in speaking proficiency. In J. Firth (ed.), Measuring spoken proficiency (pp. 1-6). Washington, DC: Georgetown University Press.
2. American Council on the Teaching of Foreign Languages. (1999). ACTFL proficiency guidelines-speaking: Revised 1999. Hastings-on-Hudson, NY: ACTFL Materials Center.
3. Baayen, R. H., Piepenbrock, R., & Gulikers, L. (Eds.) (1995). The CELEX Lexical Database (CD-ROM). Philadelphia, PA: Linguistic Data Consortium.
4. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press.
5. Balogh, J., Bernstein, J., Cheng, J., & Townshend, B. (2007). Automated evaluation of reading accuracy: assessing machine scores. Proceedings of The International Speech Communication Association Special Interest Group on Speech and Language Technology in Education (SLaTE), Farmington, PA.
6. Bejar, I. I. (1985). A preliminary study of raters for the test of spoken English (ETS RR-85-5). Princeton, NJ: Educational Testing Service.
7. Bernstein, J. (1999). PhonePass Testing: Structure and Construct. Menlo Park, CA: Ordinate Corporation.
8. Bernstein, J., DeJong, J., Pisoni, D., & Townshend, B. (2000). Two experiments in automated scoring of spoken language proficiency. Proceedings of InSTILL (Integrating Speech Technology in Language Learning, Dundee, Scotland.
9. Bernstein, J., van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing. 27 (3), 355-377. DOI: 10.1177/0265532210364404
10. Brown, C., Snodgrass, T., Kemper, S. J., Herman, R., & Covington, M. A. (2008). Automatic measurement of propositional idea density from part-of-speech tagging. Behavior Research Methods, 40(2), 540-545.
11. Brown, R. (1958). How shall a thing be called?. Psychological Review, 65, 14-21.
12. Burstein, J., Kukich, K., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., Lu, C., Nolan, J., Rock, D., & Wolff, S. (1998). Computer analysis of essay content for automated score prediction: a prototype automated scoring system for GMAT analytical writing assessment (ETS RR-98-15). Princeton, NJ: Educational Testing Service.
13. Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1-47.
14. Adams, M. L. (1980). Five co-occurring factors in speaking proficiency. In J. Firth (ed.), Measuring spoken proficiency (pp. 1-6). Washington, DC: Georgetown University Press.
15. American Council on the Teaching of Foreign Languages. (1999). ACTFL proficiency guidelines-speaking: Revised 1999. Hastings-on-Hudson, NY: ACTFL Materials Center.
16. Baayen, R. H., Piepenbrock, R., & Gulikers, L. (Eds.) (1995). The CELEX Lexical Database (CD-ROM). Philadelphia, PA: Linguistic Data Consortium.
17. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press.
18. Balogh, J., Bernstein, J., Cheng, J., & Townshend, B. (2007). Automated evaluation of reading accuracy: assessing machine scores. Proceedings of The International Speech Communication Association Special Interest Group on Speech and Language Technology in Education (SLaTE), Farmington, PA.
19. Bejar, I. I. (1985). A preliminary study of raters for the test of spoken English (ETS RR-85-5). Princeton, NJ: Educational Testing Service.
20. Bernstein, J. (1999). PhonePass Testing: Structure and Construct. Menlo Park, CA: Ordinate Corporation.
21. Bernstein, J., DeJong, J., Pisoni, D., & Townshend, B. (2000). Two experiments in automated scoring of spoken language proficiency. Proceedings of InSTILL (Integrating Speech Technology in Language Learning, Dundee, Scotland.
22. Bernstein, J., van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing, 27 (3), 355-377. DOI: 10.1177/0265532210364404
23. Brown, C., Snodgrass, T., Kemper, S. J., Herman, R., & Covington, M. A. (2008). Automatic measurement of propositional idea density from part-of-speech tagging. Behavior Research Methods, 40(2), 540-545.
24. Brown, R. (1958). How shall a thing be called?. Psychological Review, 65, 14-21.
25. Burstein, J., Kukich, K., Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., Lu, C., Nolan, J., Rock, D., & Wolff, S. (1998). Computer analysis of essay content for automated score prediction: a prototype automated scoring system for GMAT analytical writing assessment (ETS RR-98-15). Princeton, NJ: Educational Testing Service.
26. Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1-47.
27. Iwashita, N., Brown, A., McNamara, T.F., & O'Hagan, S. (2008). Assessed levels of second language speaking proficiency: How distinct?. Applied Linguistics, 29(1), 24-49.
28. Landauer, T., McNamara, D. S., Dennis, S., & Kintsch, W. (2007). Handbook of latent semantic analysis. Mahwah, NJ: Erlbaum.
29. Leacock, C., & Chodorow, M. (2003). C-rater: scoring of short-answer questions. Computers and the Humanities 37(4), 389-405.
30. Louwerse, M. M. (2001). An analytic and cognitive parameterization of coherence relations. Cognitive Linguistics, 12, 291-315.
31. Maas, H. D. (1972). Zusammenhang zwischen Wortschatzumfang und La¨nge eines Textes. Zeitschrift fu¨r Literaturwissenschaft und Linguistik, 8, 73-79.
32. Malvern, D. D. Richards, B. J., Chipere, N., & Duran, P. (2004). Lexical diversity and language development: Quantification and assessment. Houndmills, UK: Palgrave Macmillan. DOI: 10.1057/9780230511804
33. McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42, 381-392. DOI: 10.3758/BRM.42.2.381
34. McNamara, D.S. (2001). Reading both high and low coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 55, 51-62.
35. McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). The linguistic features of quality writing. Written Communication, 27(1), 57-86.
36. McNamara, D. S., & Graesser, A. C. (2012). Coh-Metrix: An automated tool for theoretical and applied natural language processing. In P.M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 188-205). Hershey, PA: IGI Global.
37. McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better?. Text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14, 1-43.
38. McNamara, T. F. (1990). Item response theory and the validation of an ESP test for health professionals. Language Testing, 7(1), 52-75.
39. Miller, G. A, Beckwith, R., Fellbaum, C., Gross, D. & Miller, K. (1990). Five papers on WordNet. Cognitive Science Laboratory, Princeton University, No. 43.
40. Munro, M., & Derwing, T. M. (1998). The effects of speaking rate on listener evaluations of native and foreign-accented speech. Language Learning, 48, 159-182.
41. Munro, M., & Derwing, T. M. (2001). Modeling perceptions of the accentedness and comprehensibility of L2 speech. Studies in Second Language Acquisition, 23, 451-468.
42. Munro, M. J., Derwing, T. M., & Sato, K. (2006). Salient accents, covert attitudes: Consciousness-raising for pre-service second language teachers. Prospect 21(1), 67-79.
43. Murphy, G. L. (2004). The big book of concepts. Cambridge, MA: MIT Press.
44. Murphy, K. R., Anhalt, R. L. (1992). Is halo error a property of the raters, ratees, or the specific behaviors observed?. Journal of Applied Psychology, 72, 494-500.
45. Norris, J., & Ortega, L. (2009). Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics, 30(4), 555-578.
46. Paivio, A. (1965). Abstractness, imagery, and meaningfulness in paired-associate learning. Journal of Verbal Learning and Verbal Behavior, 4, 32-38.
47. Pennebaker, J. W., Francis, M. E., and Booth, R. J. (2001). Linguistic inquiry and word count (LIWC): LIWC2001. Mahwah, NJ: Lawrence Erlbaum Associates.
48. Reid, J. (1990). Responding to different topic types: A quantitative analysis from a contrastive rhetoric perspective. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 191-210). Cambridge, UK: Cambridge University Press.
49. Toglia, M.P., & Battig, W.R. (1978). Handbook of semantic word norms. New York, NY: Erlbaum.
50. Shin, S. K.. (2005). Did they take the same test? Examinee language proficiency and the structure of language tests. Language Testing, 22(1) 31-57 DOI: 10.1191/0265532205lt296oa
51. Whitten, I. A. & Frank, E. (2005). Data Mining. San Francisco, CA: Elsevier. DOI: 10.1093/bioinformatics/bth261
52. Xi, X., Higgins, D., Zechner, K., & Williamson, D. M. (2008). Automated Scoring of Spontaneous Speech Using SpeechRater (SM) v1.0. Educational Testing Service, Research Report RR-08-62, Princeton, NJ.
53. Xi, X. (2007). Evaluating analytic scoring the TOEFL Academic Speaking Test (TAST) for operational use. Language Testing, 24(2) 251-286. DOI: 10.1177/0265532207076365
54. Zechner, K., Higgins, D., Xi, X., & Williamson, D. M. (2009). Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10), 883-895. DOI: 10.1016/j.specom.2009.04.009
55. Zwaan, R. A., Langston, M. C., & Graesser, A. C. (1995). The construction of situation models in narrative comprehension: An event-indexing model. Psychological Science, 6, 292-297.