What automated analyses of corpora can tell us about students' writing skills

Journal of Writing Research

Volumn 2, Issue 2, 2010, Pages 151-177

  Deane, Paul ^a   Quinlan, Thomas ^a  

^a EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Automated essay scoring; Corpus analysis; Writing assessment

Indexed keywords

EID: 85015107532     PISSN: 20301006     EISSN: 22943307     Source Type: Journal    
DOI: 10.17239/jowr-2010.02.02.4     Document Type: Article

References (77)

1. Attali, Y., & Burstein, J. (2006a). Automated essay scoring with e-rater v.2. Journal of Technology, Learning, and Assessment, 4(3). Available from http: //www.jtla.org.
2. Attali, Y., & Burstein, J. (2006b). Automated essay scoring with E-rater V. 2.0. The Journal of Technology, Learning, and Assessment, 4(3), 13-18.
3. Attali, Y., & Powers, D. (2008). A developmental writing scale. Princeton, NJ: Educational Testing Service.
4. Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Lawrence Erlbaum Associates.
5. Biber, D. (1986). Spoken and written textual dimensions in English: Resolving the contradictory findings. Language, 62(2), 384-414. doi: 10.2307/414678
6. Biber, D. (1988). Variation across speech and writing. Cambridge: Cambridge University Press. doi: 10.1017/CBO978051 1621024
7. Biber, D. (1995a). Dimensions of register variation: A cross-linguistic comparison. Cambridge: Cambridge University Press. doi: 1 0.1017/CBO9780511519871
8. Biber, D. (1995b). Variation across speech and writing. Cambridge: Cambridge University Press.
9. Biber, D., Conrad, S., & Reppen, R. (1998). Corpus linguistics: Investigating language structure and use. Cambridge: Cambridge University Press. doi: 1 0.1017/CBO978051 1804489
10. Biber, D., Conrad, S., Reppen, R., Byrd, P., Helt, M., Clark, V., et al. (2004). Representing language use in the university: Analysis of the TOEFL 2000 spoken and written academic language corpus (ETS TOEFL Monograph Series No. MS-25). Princeton, NJ: Educational Testing Service.
11. Biber, D., Johansson, S., Leech, G., Conrad, S., & Finegan, E. (1999). Longman grammar of spoken and written English. Harlow, Essex: Pearson.
12. Bos, C. S., Anders, P. L., Swanson, H. L., & Keogh, B. K. (1990). Toward an interactive model: Teaching text-based concepts to learning disabled students. In B. Keogh & H. L. Swanson (Eds.), Learning disabilities: Theoretical and research issues (pp. 247-261). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
13. Breland, H. M., Camp, R., Jones, R. J., Morris, M. M., & Rock, D. A. (1987). Assessing writing skill (No. 0-87447-280-286). New York, NY: College Entrance Examination Board.
14. SM Online Essay Evaluation: An application for automated evaluation of student essays. Paper presented at the Fifteenth Annual Conference on IAAAI'03, Acapulco, Mexico.
15. Burstein, J., & Higgins, D. (2005, July). Advanced capabilities for evaluating student writing: Detecting off-topic essays without topic-specific training. Paper presented at the International Conference on Artificial Intelligence in Education, Amsterdam, Netherlands.
16. Burstein, J., Kukich, K., Wolff, S., Lu, C., & Chodorow, M. (1998, April). Computer analysis of essays. Paper presented at the Proceedings of the NCME Symposium on Automated Scoring, San Diego, CA.
17. Burstein, J., Kukich, K., Wolff, S., Lu, C., Chodorow, M., Braden-Harder, L., et al. (1998). Automated scoring using a hybrid feature identification technique. In C. Boitet & P. Whitelock (Eds.), Proceedings of the Annual Meeting of the Association of Computational Linguistics: Vol. 1 (pp. 206-210). Morristown, NJ: Association for Computational Linguistics.
18. Burstein, J., & Marcu, D. (2000). Benefits of modularity in an automated essay scoring system. In Proceedings of the workshop on using toolsets and architectures to build NLP systems, 18th international conference on computational linguistics. Luxembourg: Association for Computation Linguistics.
19. Burstein, J., & Marcu, D. (2003). A machine learning approach for identification thesis and conclusion statements in student essays. Computers & the Humanities, 37(4), 455-467. doi: 10.1023/A: 1025746505971
20. Burstein, J., Marcu, D., Andreyev, S., & Chodorow, M. (2001, July). Towards automatic classification of discourse elements in essays. Paper presented at the Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, Toulouse, France.
21. Chenoweth, N., & Hayes, J. R. (2001). Fluency in writing. Written Communication, 18(1), 80-98. doi: 10.1177/0741088301018001004
22. Chodorow, M., & Leacock, C. (2000, April 29-May 4). An unsupervised method for detecting grammatical errors. Paper presented at the Proceedings of the First Conference on North American chapter of the Association for Computational Linguistics, Seattle, WA.
23. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Harcourt Brace Jovanovich.
24. Crowhurst, M. (1980). Syntactic complexity and teachers' quality ratings of narrations and arguments. Research in the Teaching of English, 14(3), 223.
25. Cumming, A., Kantor, R., Baba, K., Eouanzoui, K., Usman, E., & James, M. (2006). Analysis of discourse features and verification of scoring levels for independent and integrated prototype written tasks for the new TOEFL. Princeton, NJ: Educational Testing Service.
26. Dale, E., & Chall, J. S. (1948). A formula for predicting readability. Educational Research Bulletin, 27(1), 11-28.
27. Dale, E., & Tyler, R. W. (1934). A study of the factors influencing the difficult of reading materials for adults of limited reading ability. The Library Quarterly, 4 (3), 384-412. doi: 10.1086/613490
28. Deane, P., Sheehan, K., Sabatini, J., Futagi, Y., & Kostin, I. (2006). Differences in text structure and its implications for assessment of struggling readers. Scientific Studies of Reading, 10(3), 257-275. doi: 10.1207/s1532 799xssr1 003_4
29. Diederich, P. B. (1974). Measuring growth in English. Urbana, IL: National Council of Teachers of English.
30. Diederich, P. B., French, J. W., & Carlton, S. T. (1961). Factors in the judgement of writing quality (ETS Research Bulletin No. RB-61-15). Princeton, NJ: Educational Testing Service.
31. Elliot, N. (2005). On a scale: A social history of writing assessment in America. New York: Peter Lang.
32. Englert, C. S., Hiebert, E. H., & Stewart, S. R. (1988). Detecting and correcting inconsistencies in the monitoring of expository prose. The Journal of Educational Research, 81(4), 221-227.
33. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221-233. doi: 10.1037/h0057532
34. Flower, L., & Hayes, J. (1981). A cognitive process theory of writing. College Composition and Communication, 32 (4), 365-387. doi: 10.2307/356600
35. Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998). The measurement of textual coherence with latent semantic analysis. Discourse Processes, 25(2), 285-307. doi: 10.1080/01638539809545029
36. Foltz, P., Laham, D., & Landauer, T. K. (1999a). The intelligent essay assessor: Applications to educational technology [Electronic Version]. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1. Retrieved June 30, 2003 from http: //imej.wfu.edu/articles/1999/2/04/printver.asp.
37. Foltz, P. W., Laham, D., & Landauer, T. K. (1999b). Automated essay scoring: Applications to educational technology. In B. Collis & R. Oliver (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 1999 (pp. 939-944). Chesapeake, VA: AACE.
38. French, J. W. (1961). Schools of thought in judging excellence of English themes. Reprint from the Proceedings of the Invitational Conference on Testing Procedures, Princeton, NJ: Educational Testing Service, 1962.
39. Gansle, K. A., VanDerHeyden, A. M., Noell, G. H., Resetar, J. L., & Williams, K. L. (2006). The technical adequacy of curriculum-based and rating-based measures of written expression for elementary school students. School Psychology Review, 35(3), 435-450.
40. Godshalk, F. I., College Entrance Examination Board, & et al. (1966). The measurement of writing ability. New York: College Entrance Examination Board.
41. Hayes, J. R., & Flower. (1980). Identifying the organization of writing processes. In L. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 3-30). Hillsdale, NJ: Lawrence Erlbaum Associates.
42. Higgins, D., Burstein, J., Marcu, D., & Gentile, C. (2004, May). Evaluating multiple aspects of coherence in student essays. Paper presented at the Proceedings of the 2004 HLT/NAACL, Boston, MA.
43. Hunt, K. W. (1970). Syntactic maturity in schoolchildren and adults. Monographs of the Society for Research in Child Development, 35(1), 1-67. doi: 10.2307/1 165818
44. Huot, B. (1993). The influence of holistic scoring procedures on reading and rating student essays. In B. Huot & M. Williamson (Eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations (pp. 206-236). Cresskill, NJ: Hampton press.
45. Huot, B. (1996). Computers and assessment: Understanding two technologies. Computers and Composition, 13 (2), 231-243. doi: 10.1016/S8755-4615(96)90012-2
46. Kellogg, R. T. (1987). Effects of topic knowledge on the allocation of processing time and cognitive effort to writing processes. Memory & Cognition, 15(3), 256-266. doi: 10.3758/BF03197724
47. Kellogg, R. T. (1996). A model of working memory in writing. In C. M. Levy & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 57-71). Hillsdale, NJ England: Lawrence Erlbaum Associates, Inc.
48. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211-240. doi: 10.103 7/0033-295X.104.2.211
49. Lee, Y. W., Gentile, C., & Kantor, R. (2008). Analytic scoring of TOEFL CBT essays: Scores from humans and e-rater. Princeton, NJ: Educational Testing Service.
50. Lively, B. A., & Pressey, S. L. (1923). A method for measuring the ''vocabulary burden'' of textbooks. Educational Administration and Supervision, 9, 389-398.
51. Loban, W. (1976). Language development: Kindergarten through grade twelve (No. NCTE Committee on Research Report No. 18). Urbana, IL: National Council of Teachers of English.
52. Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246-276. doi: 10.1191/0265532202lt230oa
53. McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational Psychology Review, 8(3), 299-325. doi: 10.1007/BF01464076
54. McNamara, T. F. (1990). Item response theory and the validation of an ESP test for health professionals. Language Testing, 7(1), 52-76. doi: 10.1 177/026553229000700105
55. Moran, M. R. (1981). A comparison of formal features of written language of learning disabled, low-achieving and achieving secondary students. Lawrence,KN: Kansas University.
56. Nodine, B. F., Barenbaum, E., & Newcomer, P. (1985). Story composition by learning disabled, reading disabled, and normal children. Learning Disability Quarterly, 8 (3), 167-179. doi: 10.2307/1510891
57. Ojemann, R. H. (1934). The reading ability of parents and factors associated with reading difficulty of parent education materials. University of Iowa Studies: Child Welfare, 8, 9-32.
58. Page, E. B. (1966, October). Grading essays by computer. Paper presented at the Progress Report Invitational Conference on Testing Problems, New York, NY.
59. Page, E. B. (2003). Project essay grade: PEG. In M. D. Shermis & J. C. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43-54). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
60. Patty, W. W., & Painter, W. I. (1931). Technique for measuring the vocabulary burden of textbooks. Journal of Educational Research, 24, 127-134.
61. Poteet, J. A. (1979). Characteristics of written expression of learning disabled and non-learning disabled elementary school students. Diagnostique, 4(1), 60-74.
62. Powers, D. E., Burstein, J., Chodorow, M., Fowles, M. E., & Kukich, K. (2001). Stumping e-rater: Challenging the validity of automated essay scoring (ETS Research Rep. No. RR-01-03; GRE NO; 98-08bP). Princeton, NJ: Educational Testing Service.
63. Quinlan, T. (2004). Speech recognition technology and students with writing difficulties: Improving fluency. Journal of Educational Psychology, 96(2), 337-346. doi: 10.103 7/0022-0663.96.2.337
64. Quinlan, T., Higgins, D., & Wolff, S. (2009). Evaluating the construct coverage of the e-rater scoring engine. Princeton, NJ: Educational Testing Service.
65. Ransdell, S., & Levy, C. M. (1996). Working memory constraints on writing quality and fluency. In C. M. Levy & S. Ransdell (Eds.), The science of writing (pp. 93-105). Mahwah, NJ: Lawrence Erlbaum Associates.
66. Reppen, R. (2001). Register variation in student and adult speech and writing. In S. Conrad & D. Biber (Eds.), Variation in English: Multi-dimensional studies (pp. 187-199). Essex: Pearson ESL.
67. Salton, G., Yang, C., & Wong, A. (1975). A vector space model for automatic indexing. Communications of the ACM, 18(1 1), 613-620. doi: 10.1 145/361219.361220
68. Sheehan, K., Kostin, I., Futagi, Y., & Sabatini, J. (2007). Reading level assessment for informational and literary texts. Paper presented at the 29th Annual conference of the Cognitive Science Society, Nashville, TN.
69. Sheehan, K. M., Kostin, I., Deane, P., Hemat, R., Zuckerman, D., & Futagi, Y. (2006). Inside sourcefinder: Predicting the acceptability status of candidate reading comprehension source documents (ETS Research Rep. No. RR-06-24). Princeton, NJ: Education Testing Service.
70. Sheehan, K. M., Kostin, I., & Futagi, Y. (2007, October). SourceFinder: A construct-driven approach for locating appropriately targeted teading comprehension source texts. Paper presented at the SLaTE Workshop on Speech and Language Technology in Education ISCA Tutorial and Research Workshop, Farmington, PA
71. Spandel, V., & Stiggins, R. J. (1990). Creating writers: Linking assessment and writing instruction (2nd ed.). London: Longman.
72. Stewart, M. F., & Grobe, C. H. (1979). Syntactic maturity, mechanics of writing, and teachers' quality ratings. Research in the Teaching of English, 13, 207-221 5.
73. Strunk, W. (2000). The elements of style. Boston, MA: Allyn & Bacon.
74. Vogel, M., & Washburne, C. (1928). An objective method of determining grade placement of children's reading material. Elementary School Journal, 28(5), 373-381. doi: 10.1086/456072
75. Will, U., Nottbusch, G., & Weingarten, R. (2006). Linguistic units in word typing: Effects of word presentation modes and typing delay. Written Language and Literacy, 9 (1), 153-176. doi: 10.1075/wll.9.1.10wil
76. Witte, S. P., Daly, J., & Cherry, R. (1986). Syntactic complexity and writing quality. In D. McQuade (Ed.), The territory of language (pp.150-164). Carbondale: Southern Illinois University Press.
77. Wolfe-Quintero, K., Inagaki, S., & Kim, H. (1998). Second language development in writing: Measures of fluency, accuracy and complexity. Honolulu, HI: University of Hawaii: Second Language Center.